Interpret Your Decision: Logical Reasoning Regularization for Generalization in Visual Classification
Authors: Zhaorui Tan
Xi Yang
Qiufeng Wang
Anh Nguyen
Kaizhu Huang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Domain generalization
generalized category discovery
image classification
logical reasoning
Primary area: machine_vision

Abstract: Vision models excel in image classification but struggle to generalize to unseen data, such as classifying images from unseen domains or discovering novel categories. In this paper, we explore the relationship between logical reasoning and deep learning generalization in visual classification. A logical regularization termed L-Reg is derived which bridges a logical analysis framework to image classification. Our work reveals that L-Reg reduces the complexity of the model in terms of the feature distribution and classifier weights. Specifically, we unveil the interpretability brought by L-Reg, as it enables the model to extract the salient features, such as faces to persons, for classification. Theoretical analysis and experiments demonstrate that L-Reg enhances generalization across various scenarios, including multi-domain generalization and generalized category discovery. In complex real-world scenarios where images span unknown classes and unseen domains, L-Reg consistently improves generalization, highlighting its practical efficacy.

Interpret Your Decision: Logical Reasoning Regularization for Generalization in Visual Classification
Zhaorui Tant?2, 2,n Xi Yang"', Qiufeng Wang', Anh Nguyen², Kaizhu Huang
Xi'annJiaotong Liverpool University
University of Liverpool
Duke Kunshan University
Abstract
Vision models excel in image classification but struggle to generalize to unseen data, such classifying images from unseen domains discovering novel categories. In this paper, we explore the relationship between logical reasoning and deep learning generalization in visual classification. A logical regularization termed I.-Reg is de- rived which bridges logical analysis framework to image classification. Our work reveals that L-Reg reduces the complexity of the model in terms of the feature dis- tribution and classifier weights. Specifically, we unveil the interpretability brought by L-Reg, it enables the model to extract the salient features, such faces persons, for classification. Theoretical analysis and experiments demonstrate that L-Reg enhances generalization across various scenarios, including multi-domen domain generalization and generalized category discovery. In complex real-world scenar- ios where images span unknown classes and unseen domains, I.-Reg consistently improves generalization, highlighting its practical efficacy.
Introduction
L-Reg
One critical challenge in visual classi- fication models is their ability to gen- eralize effectively unseen samples or unknown classes. For instance, a model trained on real images of vari-
Unseen domain
Seen domain
ous animals should ideally classify an- imal sketches accurately (referred to as molti-domain generalization classi-
With L-Reg
fication (20 35
25 B7 50]) or
discover novel categories not present in the training set (referred as gen-
Figure 1: GradCAM 455 visualizations for the unknown class 'person' across seen and unseen domains of the GMDG baseline with Le regularization that is trained without and with L-Reg, respectively. Both experiments share the same hyper-paramettes,, except the latter uses the L-Reg.
eralized category discovery [54] [6]). These problems are prevalent in real- world scenarios, where training data-
target pairs are usually insufficient, and labeling is time-consuming so that not every data is paired with a label. Meanwhile, test data is likely to contain shifts in both data and targets, making it essential to propose methods that generalize to border scenarios.
Regularization terms, such as L2 regularization leading to weight decay, are commonly employed during training improve a model's generalization capabilities. However, the L, regularization
"Corresponding authors. <Xi. Yang00 @ @xiilu.edu.en; kaizhu, huang wukkekunsaan.dd..cc..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

is parametric-base rather than sample-based, which may lead to ambiguous interpretability [58]. As illustrated in Fig. II the model trained solely with L2 regularization exhibits low interpretability.
Other regularization terms [57 58 59 attempt to improve the interpretability of deep learning models for sequential signals rather than vision, whereas [39] proposes a regularization term to enhance interpretability for robustness in visual classification models rather than generalization. Drawing inspiration from logical reasoning has shown promise for better generalization and interpretability in various tasks. Current work unveils the effectiveness of logical reasoning in generalization tasks, such as boosting performance in length generalization T 2 60) and abstract symbol relational reasoning [10] (6) (e.g., mathematical solving and psychological tests). Several efforts, such as [6], explore the explicit antropy-based logical explanations of neural networks for image classification, confirming the presence and interpretability of logical reasoning within visual tasks. Yet. there are limited studies tackling the generalization of visual classification tasks through the lens of logical reasoning.
This paper studies two pivotal questions corresponding to the above: 1) How does logical reasoning relate to visual tasks such as image classification? 2) How can we derive logical reasoning-base regularization term to benefit generalization? To achieve these, we correlate the image classification procedure in computer vision with the framework of logic studies [4, positing that training an image classifier involves learning a good general logical relationship between images and labels via an encoder. This good general logic is attained when the semanties generated by the encoder and classifier can be combined to form atomic formulas. Our exploration leads to the introduction of a cample-baedd Logical regularization term named L-Reg. We reveal that L-Reg efficiently reduces the complexity of the model from two aspects: 1) L-Reg leads to a balanced feature distribution in the semantic space; 2) L-Reg reduces the number of weights with extreme values in the classifier.
Intuitively, the complexity reduction achieved by L-Reg stems from its ability to filter out redundant features or semantics, focusing instead on the minimal yet sufficient semantics for classification defined as semantic support in Definition 3.2 where the interpretability also emerges. This filtering feature benefits the generalization when there is domain shift in data where the domain-dependent features are ignored for classification. Moreover, it further promotes generalization when unlabeled data from the unknown classes is present. If such data lacks the semantic support associated with known classes, it is then classified as belonging to an unknown class, and its corresponding semantic supports are extracted. These capabilities equip L-Reg with explicit interpretability. As Fig. [ shows, with L.Reg, the model can identify the unknown class 'person', and pinpoint faces which are the crucial features for classifying this category. In contrast, the model trained solely with L2 (without L-Reg) focuses on the ambiguous features for classification.
Rigorous theoretical analysis and experimental results validate that L-Reg yields better generalization across diverse scenarios. Specifically, L-Reg facilitates better performance under the aforementioned multi-domain generalization and generalized category discovery tasks, whose settings presented in Fig. (b).). Furthermore, to evaluate L-Reg's ' robustness, we introduce a more complex real- world scenario, shown in Fig. 2 (c), where unlabeled images may not only belong to unknown classes, but also originate from unseen domains. Even in this challenging context, L.-Re is still able to consistently demonstrate notable improvements in generalization, underscoring its practical utility and effectiveness. Our code is available at httph:: :/gittub com/ haamm//hhoriitttanaaea RegNeer IPS24
2 Preliminaries and generalization settings for visual classification
Consider paired (X,Y) (X), and (X,, Yo)= ~ (2, denote all sets of inputs and labels, seen paired subsets of (X, Y). and unseen paired subsets of (X, Y), respectively. Note that X,, Y. may be accessible for the model separately, but their pairing relationships are not ac- cessible. Let D denote the possible domains, with D., D. C D representing the seen and unseen do- mains. classification tasks, an encoding function g(x) + 2 E M is commonly introduced to map X into the latent feature set Z, where each latent feature has M dimensions. A predictor h(Z) + Y e I
(X,3). (Xs; Y.)
(0L Molli-domalt
(b) Generaliaed disconety
101. Mutt--omminn gensrulization Ganctolized Sallagey discovery
K. 1| P.jo.
A.I.. 7.P. K,I. TJl.
Z.Io, TT.0P.
Ax
V.j0. Z.j..
Treget-shin
X,10,
Sete availiable for training Sete unavaliable for trriiinngg evaluation
K,10,
9,19
All-hiit
Figure 2: Diagrams of different generaliza- tion settings in visual classification tasks.
maps Z to predictions Y. where K denotes

---

All classss
Khown classes
Unknown dissses
Known
1.
Unknown
Wintut-faa
Rey
Without L-Reg
Normalized weight
Heatmap classifer's weights
(b). Distribution of values of classifer's weights under classes
Figure 3: Visualizations of classifiers' weights form models trained using GMDG on PACS dataset without and with L.-Reg under mDG+GCD setting, respectively. Both experiments share the same hyper-parameters using Regnety- backhone, except the latter uses additional L-Regg
the number classes and the dimensions of predictions. P-)) and H(-) symbolize probability and entropy, respectively. This paper discusses two typical cases for generalization in image classification tasks: (1) Data-shift generalization: X, and X,, have distribution shifts, such as multi-domain generalization (mDG); and (2) Target-shift generalization: Y, and Y, have distribution shifts, which stands for tasks like generalized category discovery (GCD). We additionally explore challenging scenario called All-shift generalization: both X, and X.. Y, and Y.. have distribution shifts, which is a combination of mDG and GCD tasks (mDG + GCD). The following lists the detailed settings for generalization. Please refer to Fig, for brief diagrams.
Data-shift generalization: Problem setting for mDG. Illustrated in Fig. 2 (a). mDG (9] intends to generalize well to unseen domains having the objective of min H (X Y. D.) and expecting the model be generalized to X,, when predicting Y, from the unseen domain D... In such cases, Y,
fully accessible to the model since Y, and Yu share the same domain: 3 Y: but there are shifts in X where X * Xx.
Target-shift generalization: Problem setting for GCD. GCD 54 (Fig. (b)) aims to discover possible unseen labels among unlabeled datasets X., The challenge is that the samples in X,, may belong to known classes or unknown classes: V. + Vn and probably V, nor * 0. The model should be able distinguish the samples from the known classes and cluster the samples for unknown classes simultaneously. Note that X2 is used for model training, but the relationship between X, and Y, is unseen for the model. summary, shifts exist between Y, and Y. not between X, and X
All-shift generalization: Problem setting for mDG + GCD, To explore the generalization problem further, we introduce setting that is the combination of mDG and GCD shown in Fig. 2 (c). Specifically, the model is trained on the labeled pairs (X,, Y,) and unlabeled set X,, from the seen domains D.; X., may belong known and unknown classes. Furthermore, the model is tested on X,, from the unseen domain D.., where X,, may also come from the known and unknown classes. In this setting, the model is expected to classify samples the seen classes and discover the unseen classes among unlabeled samples from seen domains and 2) generalize this ability to the samples from the unseen domain. In this scenario, X, and X,, have shifts, and so do Y, and Y..
For all aforementioned generalization settings, the objective can be summarized as minimizing the generalization loss:
Definition 2.1 (Generalization loss). Let the target model f fx,, Y): X + Y, can generalize across both seen and unseen sets X, Y. Denote its trainable form f. which is only trained on the seen sets. The generalization loss for the unseen sets defined
GLJJ, S", (X., Y.)) (xe..)E(..., (xx,
3 Logical regularization for generalization in image classification
Under the problem settings defined in Section 2 we introduce Logic regularization (L-Reg) targeting
the objective:
min E,,cs. D)] D D)},
(2)
3

---

where Y R = h o g(X) is the prediction set. The corresponding Logic regularization loss (L-Reg) defined as:
J.a (Y 2) log o (Y
Lo- Rog
M
1 Σ
3. log
where J. T Z) denotes the value the i.j position of softmax Z) and the soft-max function is applied at the last dimension. By incorporating other existing methods' losses denoted by Lmain the overall loss is formulated as:
Laul = Lmain + aL1. Reg:
(4)
with a weight a applied to balance two losses. As depicted Fig. I L-Reg plays a pivotal role in extracting crucial features for image classification, thus enhancing generalization capabilities. This beneficial outcome can be attributed to two primary factors:
Reducing classifier complexity: I.-Reg streamlines the complexity of the classifier itself, as depicted in Fig. 3 (a). Notably, the heat map of the model with L.-Reg displays fewer extremely valued weights, evidenced by the diminished presence of intense blue and red colors. This reduction implies that the classifier focuses on leveraging semantically rich and relevant features for decision-making (classification), sidelining the less relevant ones. Additionally, Fig. (b) reveals reduction in the number semantic features used to classify each class.
Balancing feature complexity: L-Reg results in more balanced distribution of features compared to the base- line, as illustrated in Fig. + This balanced distribution suggests the elimination of certain extracted semanties characterized by dominant frequencies across all sam- ples. Semantics that occur frequently across samples of- ten lack decisiveness for classification. Hence, reducing their prominence contributes to more expressive feature space and less complex feature distributions. Coupled with the reduced classifier complexity, a simplified classifier achieved through L-Reg facilitates improved generaliza- tion across various settings. Specifically, the top row also indicates the distance between the feature distributions of the known and unknown classes, which enlarged; thus, they more dividable, leading to classification improvements.
L
with 1 RReg
Figure 4: Visualizations of latent fea- tures form models trained using GMDG on PACS dataset without and with L- Reg under mDD+GCD setting using RegNetY-16G backbone, respectively.
We present -ogical-based theoretical analysis in Section 3.] and provide the derivation details of L-Reg in Section 3.2 In addition, we discuss the efficacy of L-Reg under various generalization settings in Section 4 Furthermore, L-Reg serves as a plug-and-play loss function that is compatible with most existing frameworks. We conduct experiments applying L-Reg to various established approaches across different generalization settings, as outlined in Section
3.1 Logical framework for visual classification
This part provides the connections between logical reasoning and visual classification tasks. We would like to remind readers of the framework for studying logics and link it with our practical scenarios.
Definition 3.1. Following 4 logic L is defined five-tuple in the form:
(5) where 1) Fc denotes the set of formulas formed by images and labels (X, Y); 2) Me represents different domains D of X; 3) Fc is a binary relation relating the truth of whether the formulas are true or false, which has F Mc x Fz: 4) mngg. : Fr. Mc. - Sets defines the meaning of X as determined by classifiers, where Sets indicate the class of all sets. (5) Fc symbolizes the provability relation of C, evaluating formulas formed by magc. is true or false in one possible world,
L - (Fc, Mc, Fc, mngg, H2),
such the estimation criteria. More details of L can be seen in Appendix B

---

For clarity, we specify ((x,,,)) = Dx h, F(m(x)))) as the logic formed on the given X, Y sets. With the goal for logic to generalize across a broader scenario and provide extrapolation across all possible formulas in L, good general logic L should be derived from L through the feature extractor g:
L (F(o(x.).Y)) D, Fix(..)) h ((og(())))) s.L., s.L., ((og(())))))
(6)
Importantly, as a good general logic, F(o(x.).Y.) and in L should form the atomic formulas, ie., the tuple of terms with a predicate: h o g(x) belongs/not belongs class y in domain d + Ture/False, where z, U. d E X, Y. D, which makes that (() We simply denote one atomic formula in the form of h(g(3r), y, d) mapping to binary values. Addi- tionally, F (
still holds.
in Eq. 66) can be safely omitted in the rest of the paper. Please see more details about the conditions of the good general logic in Appendix B
An additional tool is necessary convert the logic problem into a continuous form, enabling the application of machine learning algorithms. The conditional entropy-based method enables a logically sound derivation of knowledge from the provided dataset with constraints
44 Specifically, the probabilistic inference process adheres to a probabilistic version of Modus Ponens: A + B, A - B (if A then B; not A therefore not B). It is important to note that the logical propositions in probabilistic Modus Ponens are uncertain, with the conditional probability replacing the material implication interpret logical deduction through the lens of entropy.
A + B. This framework allows Therefore, for Eq. (6) which implies
3h o 9, V(z,y) y) E (X,Y), Ad E D, ho - y:
finding h o g through optimization is equivalent to
max Bx..))Ecx.YYddd P(ylg())) h.9
R + min Beweeee R, (8)
h,
where R denotes any other possible regularization.
As the logical framework for image classification takes shape, it becomes evident that the unresolved question of identifying an appropriate function g to generate suitable atomic formulas emerges as a
critical factor in ensuring the effectiveness of the overarching logic L. This paper proposes L-Reg as the regularization to ensure ((g(X.),Y.)) are formed by atomic formulas in Section 3.2
3.2 Constructing atomic formulas using L-Reg
In this part, we show the derivation details of I.-Reg the aims to ensure the formation of suitable atomic formulas, depicted in Eq. (6). As highlighted in (1, current algorithms may induce implicit biases towards unseen data, resulting in varied solutions for such data. However, expecting an algorithm to generalize effectively unseen data domains without appropriate incentivization, such as specifically designed regularization, is unreasonable, Therefore, we aim to enhance the generalization capability of models by employing a logic-based regularization approach. To this end, we introduce the concept of semantic support for image classification.
g(x), where = E Z, as a of compositions of these semantics: 2 := {2²} 1a where M is the number of dimensions or semantics. Notably, not all semantics
Definition 3.2 (Semantic support). We denote
may be useful for deduction or inference. We define the subset y of 2, extracted from the sample x ~ X, as the semantic support of x if y is sufficient for deducing the relationship between x and a y ~ y.
For instance, if the subset {z1, C is sufficient for accurate inference, the values of other semantics {= will not impact the inference process. When {²), constitutes the minimal combination of semantics required for inference, it is termed the semantic support. We denote I as the set of semantic supports of X for deducing each individual class.
Derivation of L-Reg, Regarding Eq. (6. if the semantic supports and their relationship with Y form atomic formulas, Eq. o6) holds good general logic, and the generalization would be improved. Thus, we aim to learn the latent features Z, which contain sufficient semantic supports for the deduction of Y:
Bz E 2, v) E (Z,Y), Ad E D, h(rld) +
(9)
Specifically, g(-) should meet the following:
V(T.,))) 85) E (Z,Y), Vd D, Di # - r *r,
(10)
5

---

i.e., the semantic support set for each class should be distinct. The multiple- classsseeessss classification task has that VT, [T] I M.. Under the constraints demonstrated in Eq.
9 and Eq. (10). we need to achieve
the following through optimization:
min (g D), max (Y ())) - min H -) Ag
(11)
hg
kgg
where P denotes the negation of r. i.e., the set of semantics which does not include semantic support.
Intuitively, Eq. (TT) regularizes that the model should be able to judge whether a sample belongs to class by using a minimal set of semantic supports; simultaneously. the semantic support sets are also implicitly disentangled for each class, not only for maintaining rich and useful semantics but also for enhancing the independence of deduction of each class. The actual collection of appears to be intractable during optimization. Hence, we resort to deriving its bounds. Regarding Eq. [1] its former term can be elaborated as follows:
H(Y)g(T) D) S H(YIM(Z)) D) S
D)).
(12)
where 21 is minimal semantics form 2, and E. ((Ygg(())))))))))))1)) D) is the upper-bound for ming, ( D). Therefore, minimizing EM H g(z;), D) equivalent to minimizing H(Y)g(T), D). Meanwhile, for the latter in Eq. 11), we have:
H(Y]g(T),D) 2 (()))()),D),
(13)
where HY))n==)) D) is the lower-bound for maxh.g H(Y]g(T), D). Combining the aforementioned bounds, we have the L-Reg objective as Eq. (2).
Interpretability of semantic supports roots in forming atomic formulas. The atomic formula A of the form h(g(z), ,,,)), d). Our aim is to find the good (most) general A E A for y class from which the interpretability of I.-Reg is derived. Consider A Ag E A", if Ad is more general than Ag. there will be a substitution & such that Arv = A" [52]. Av should meet A = A² E A", which infers that ? nd (cf. Eq. P for predication of y where 7 is the semantic support. Note here that the form of A is constructed for y E Y. i.e., predicate whether the sample belongs to the y class. Considering multiple classes yo: MJ E YY,i + j. it has A + A thus 7 o y (cf. Eq. (1O), which constrains that different minimal semantic supports should be used for predicting different classes. The interpretability of L-Reg is based on A compelling the model to use distinct minimal semantic supports for each class. These minimal semantic supports can be interpreted as the most critical features for efficient prediction. For example, as shown in Fig. [1 the model with L-Reg has learned the facial features of the person class (see more examples in Appendix Figs. to T2). forming the (informal) atomic formula h (has a human face, is person, d E D) -> True. Similarly, also leads to (not has a human face, is person, d E D) + False
4 L-Reg under different generalization settings
L-Reg under data-shift generalization. The task mDG endeavors to facilitate a model's ability to generalize to unseen domains by fostering invariance across seen domains [50]. In the context of mDG, the term [D] 2 in Eq. (8) typically denotes multiple domains. Traditionally, existing methods focus on minimizing domain gaps, leading to remarkable results [2] 50]. However, it is
noteworthy that even when the domain gap is effectively minimized, and |D = for the latent features can be considered, L-Reg still demonstrates its efficacy in promoting the generalization of X, from D.,.
Proposition 4.1 (Effectiveness of L-Reg in enhancing data-shift generalization.) Assume the gap across all domains is well minimized. Let I denote the target model that generalizes to the data X from the unseen domain with the lowest complexity. For a model JR ((x,, S(X.YX) trained under the data-shift generalization setting (i.e., (X,, Y.) is accessible and y, Vul. We have:
GL f",X,) S GT.JJX.YY)''"' Xw))
(14)
Please see proof details in Proposition intuitive example: In the seen domains, all cats are either black or white, while all dogs are brown. Now, imagine encountering a sample labeled 'a brown cat' from an unseen domain. Without the application of L-Reg, the model might erroneously classify it as dog. However, with L-Reg in
To illustrate Proposition 4.1 consider the following

---

Table MDG results: Comparison between the proposed and previous non-ensemble and ensemble mDG methods. The best results for each group are highlighted in bold. Improvement and degradation our approach from GMDG are highlighted red.
Test domain
PACS
VI.CS
OfficeHome
Terralncognita
Domain
Avg.
MMD
84.7a0.5
77.510.9
66.320.1
12.2+1.6
23.449.5
58.8
Mixstyle
(2:
85,2+0.3
77.9+0.5
60.4#0.3
44.0=0.7
34,0+0.1
60.3
GRo
$4.420.8
76.7
43.2a1.1
33.340
IRM 5
83.5+0.8
78.5+0.5
64.3=2.2
47.6+00.8
33.9+28
61.6
ARM [1]
85.1=0.4
77.6+0..
64.8+0.3
45.5+0.3
35, 15.54012
61.7
84.920.6
78.340.2
16.429.6
61.9
CDANN
82,6+0.9
77.5+0.1
65.8=1.3
45.8=1.6
38.340.3
62.0
DANN 20
83.6+t0.4
78 6+00
65.9#0.6
46 46.5
58.340
62.6
RSC
85,240.9
16.621.0
MTL 8
84.620.5
77.2+0.
66.420.3
45.6=1.2
40.640.1
62.9
MIDG 31
84.9+1
77.2+04
66.8+0.6
47.7+0.9
41,240.1
63.6
Fish Hb
77.8+0.3
ERM S3
84.2+0.1
77. 3+0.1
67.6+0.2
47 4+.8+
44,0H0.1
64.2
NagNet 40
86.320.2
77.810.5
68.120.1
18.62100
40.340.1
64.2
SeltReg
77,9+0.9
47.0+0.3
CORAL
86.240.3
78 78.8
68.7=0.3
47.6+1.0
41.5+0.1
64.5
mDDDI
86,220.2
79.0:0.3
69.2s004
48.1214
42,810.1
65.1
Use RegNerY-I6GF
47
as oracle model.
MIRO
23 (BCCV23)
97.420.2
79.9
80.4a0.2
58.921.3
$3.840.1
74.1
GMDG
5D (CVPR24)
97.3+0.1
82.4+0.6
80.8=0.6
60.7=1.8
54. 64.6+0.1
75.1
GMDG
L-Reg
97,4+0.2
82.4+0.0
80,9=0.5
62.9=0.9th
$5.340.0
75.8
place, the model is compelled rely on minimal semanties for classification. This means filtering out irrelevant features such as color terms, thus enabling more accurate deductions.
L-Reg under target-shift generalization. We demonstrate how L-Reg enhances generalized discov- ery in scenarios where only a subset of classes (Y,) is available for training, and there may exist an overlap between the unseen classes (Y,) and the seen classes (Y₂). denoted as Vu ny,, * 9. We define Yu/Y, as the novel classes not included in Y,, and Yu ~ Y, as the seen classes for X, classification, where [D]
Building upon Proposition 4.1 L-Reg further enhances GCG by improving the generalization performance on Y..
Proposition 4.2 (L.-Reg improves target-shift generalization). When [D] = 1, I.-Reg promotes generalization performance Yu under the target-shifi scenario.
Proof. When |D] = 1, since all Y belongs to a close set, minimizing - (Y,) D) equivalent to the following:
min (Yollg(T)) h.
min (Y.,g((T)) hgg
(15)
where Y, is the negation Y., i.e., Yu/Ys. this situation, one sample does not contain sufficient semantic support to be classified under Y., it otherwise will be assigned under Y/Y., promoting performance for both Yu,Y,, and Y, Yg. Therefore, the generalization performance on the unseen classes will be improved by L-Reg.
L-Reg under all-shift generalization. When the domain gap is sufficiently minimized and |D] = can be considered, the combination of Proposition 4.1 and Proposition 4.2 demonstrates that L-Reg enhances generalization performance on both novel classes (Y./Y,)) Y.) and seen classes (Y, ~ Y.) for X , from other domains. Our experiments validate that L-Reg, when applied in scenarios with well-minimized domain gaps, consistently improves generalization across all shifts.
5 Experiments
To validate L-Reg, three groups of experiments under the three kinds of settings conducted. Notably, all baselines we used already incorporate the L2 regulation in the form of weight decay. We also compare other commonly used regularization terms, such independence sparsity regularization on Z. More results Appendix [ indicate that our L--eg also surpasses them.
5.1 Experiments on mDG
Experimental settings. We operate on the DomainBed suite (2 and leverage standard leave- one-out cross-validation as the evaluation protocol. We test L.-Reg with GMDG (50] on 5 real- world benchmark datasets: PACS (32. VLCS [18), OfficeHome [55), TerraIncognita Z and
7

---

Following MIRO [25 and GMDG [50], the RegNerY-I6GF backbone with SWAG pre-training used. Specifically, we train the backbone using GMDG with L-Reg. Accuracy is adopted as the evaluation metric, and the results of the averages from three trials of each experiment, with standard deviations, are presented. See Supplementary
DomainNet 42
H for more experimental details.
Results. The experimental results presented in T demonstrate the efficacy of L-Reg in improving the performance of GMDG across all datasets in mDG classification tasks. Notably, more substantial improvements are observed when the GMDG baseline achieves relatively low accuracy. These observed enhancements provide empirical support for Proposition 4.1 Please see using L-Reg with basic ERM in Appendix E. For detailed insights into each domain within each dataset, please refer to Appendix H.
Table 2: GCD results: Average results across all datasets of PIM with L-Reg, Improvements and degradation are high- lighted in red and blue, respectively.
5.2 Experiments on GCD
Experimental settings. We validate our approach through training PIM additionally with L-Reg. Six image datasets are adopted to validate the feasibility of our proposed L-
Average
All
Known
Unknown
Reg with PIM compared to other competitors, including three generic object recognition datasets, CIFARIO [29].
K
44.7
46.0
Rank
22 (TPAMI-21)
54.6
UND-
[9 (TCVV-21)
51.2
74.5
CIFAR100 (29) and ImageNet-100 100
17]: two fine-grained
ORCA
(ICLR-22)
51.3
ORCA -VTR16
56.7
65.6
49.9
datasets CUB [56] and Stanford Cars (28; and the long-
(CVPR-22)
71.8
RIM
(NeurtP-- 10)
62.0
72.5
53.4
tail dataset Herbarium19 [4]. Following prior works [54] 16. we use the proposed accuracy metric from (54) of all classes, known classes, and unknown classes for evaluation. Please see detailed description of the exper- imental setup in Appendix H.2
TIM
11 (NeudIPS-20)
02.7
72.6
56.4
PIM
16
(KCCV-25)
67.4
79.3
59.9
PIM
L. Keg
68.8
79.0
62.7
Table 3: MDG+GCD results: Averaged accuracy scores for all, known and un- known classes across all five datasets. Improvements and degradation are high- lighted in red and blue respectively.
Results. The average results across all datasets for uti- lizing L-Reg with PIM are presented in Table [ while detailed -ppesct-specific information is available in Ap- pendix|H Table 17
The results highlight that L-Reg con- sistently increases the accuracy of all unknown classes across all datasets, thus confirming the validity of Proposi- tion 4.2 However, is notable that L-Reg may marginally compromise the performance of known classes, as it re-
Method
gap
Known
Unknown
ERM
Not
44.69
59.33
23.34
.L-Reg minimized
45.50
61.43
21.63
Imp.
2,09
-1.91
PIM
46.35
26.90
+L-Reg minimized
47.27
60.83
26.34
Imp.
0.32
0.48
-0:57
duces the size of semantic support for deducing Y, thereby reducing the information available for known classifica- tion. Nevertheless, this compromise is deemed acceptable given the significant improvements observed for the un-
MIRO
Not sulticcanlly
25.79
LL
minimized
52.11
71.26
26.49
Imp.
2:44
2.39
0.71
GMDG
47.94
68.75
21.68
+I-Rog
Minimized
51.94
69.87
27.68
Imp.
4:00
1.12
7.01
known classes.
5.3 Experiments on mDG + GCD
Experimental settings. We utilize datasets designed for mDG tasks to conduct mDG + GCD experiments. During the training stage, only samples from seen domains are available, with half of the classes masked as unknown, and only their unlabeled data are utilized. Notably, even though all the unlabeled data originates from unknown classes during training, this prior knowledge is not assumed or constrained, aligning the setting with GCD. Similar to mDG, we adopt the leave-one-out cross- coossvalidatiin method. This entails testing each domain in each dataset as the unseen domain. The performance is tested on unseen domains by employing GCD metrics. To validate L-Reg's s efficacy comprchensively, we re-implement four methods under the mDG GCD setting, testing them both with and without L-Reg. The four methods include ERM, PIM. MIRO, and GMDG. ERM serves as the baseline approach without additional regularization, while PIM maximizes information without minimizing domain gaps. MIRO and GMDG focus on minimizing domain gaps, with GMDG offering comprehensive approach in this regard. It is worth noting that PIM has been re-implemented. For further experimental details, please refer to Appendix H.3
Results. The averaged results across all unseen domains of all datasets are summarized in Table 3
For detailed breakdown of results for each domain in each dataset, please refer to Appendix H.
As discussed in Proposition 4.1 and Proposition a noticeable trend is observed wherein, as the
8

---

domain gap is gradually minimized, the improvements for unknown classes increase, with the best results achieved using GMDG with L-Reg.
Unknown class
L.-Reg forms atomic formulas and improves in- terpretability. Furthermore, Fig. provides visual insights into the behavior of models trained with L- Reg. Evidently, these models tend focus on min- imal semantics sufficient for class distinctions. For the known classes, the efficacy of L-Reg can be intu-
Photo
Scren domains
cartoon
itively understood as extracting the minimal semantic supports for given class label. For instance, the presence of a guitar's 's fingerboard, even in unseen domains, helps classify a sample as belonging the guitar category, whose informal forms can be denoted as hhhs fingerboard, guitar,
Sketch
D) -> True and hnno has fingerboard, is guitar, d D) - False. For all known classes, samples with these minimal semantic supports are recognized accordingly. In con- trast, a sample lacks these minimal supports for any known class, it is very likely categorized as known class. This behavior stems from Paper Eq. 10 which ensures A # A through constraining y * yM5. L.-Reg further enhances the model's abil- ity to identify minimal supports for unknown classes by filtering out co-covariant features associated with other classes and thus generalizing to unseen domains. Therefore, the very interpretable features for unknown classes from unseen domains can be extracted using L-Reg. Fig. 5 (right side) demonstrates that the model with L-Reg can even extract facial features for the unknown person class and can generalize this the unseen domain. Similarly, here we obtain (informal) atomic formulas as (has a face, is person, d E D) + True, h(not has face, person, E D) False.
Art g
Useen domain
Without L-Reg With L-Reg
Without L-Reg
With L-Reg
Figure 5: GradCAM visualizations of GMDG trained without and with L-Reg. The seen, unseen domains and known, unknown classes are denoted.
However, as shown in Row 3, significant domain shifts, such as those between the sketch domain and other domains, pose challenges. Specifically, the differences between the stick-figure style of sketches of persons and figures from other domains can hinder the model's ability to cluster sketches with other domains' figures when the class label is unknown. Thus, under this cireumstance, the model may fail to extract meaningful features from those sketches. We acknowledge this limitation and will explore solutions in future work.
Table 4: Averaged results of applying L-Reg to different layers across domains in PACS.
L-Reg should applied to features from deep layers. One crucial precondition highlighted in the theoretical analysis is that L-Reg operates effectively with representation Z. where each dimension repre- sents independent semantics. The semantic features usually come from the deeper layers of the model
All
Known
Unkown
I-Keg Deep layer
67.82
91.86
31.33
L-Reg: Earlier and the deep layers
58.97
80.73
35,05
architecture [51] However, Table Table[Ishows shows that applying L-Reg to features from earlier layers, which may not necessarily represent semantics, leads degradation in performance for known classes, albeit improving performance for unknown classes. This phenomenon arises due to the potential interdependence among features from earlier layers, resulting in penalization that may hinder the capture of semantic supports essential for known classes. To ensure generalization improvements without significant compromise to the performance known classes, we advocate for applying L-Reg specifically to features extracted from deeper layers, such as the bottleneck layer. These suggest that the compromised results observed in Table structure, which fails to provide the expected semantic features.
2 could be attributed the less depth of the model
5.4 Apply I.-Reg to congestion prediction for circuit design.
Experimental settings. We also test L-Reg in Congestion prediction on the CircuitNet [5] dataset by using CircuitFormer (63) backbone. The congestion prediction is for circuit design and benefits from logical reasoning-basee approaches. All parameters, except for L-Reg, remain consistent with FircuitFormer, and we follow its metrics.

---

Results. Table 5 shows the results of predic- tion results on the CircuitNet dataset. We also include the results of Gpdl with UNet++ and Cir- cuitFormer for better comparison. Notably, the improvements brought by CircuitFormer with L-- Reg across all metrics, especially for the pearson metric can be observed. The consistent improve- ment with L-Reg across all metrics indicates L-Reg's feasibility.
Table Results of Congestion prediction: Con-
gestion prediction is proposed for circuit design.
Pearson
spearman
Kendail
Gpul with UNet++
0.6085
0.5202
0.3855
CircuitFomer (SOTA)
0.6374
0.5282
0.3935
CircuitFormer L-Reg (Ours)
0.6553
0,5289
0.3944
Related work
Logical reasoning for deep learning. Current studies focus on length generalization symbolic reasoning in the logic-based scope. For length generalization, [1] proposes the generalization to the unseen setting, theoretically verifying that commonly used models can generalize to the unseen and degree curriculum promotes the generalization ability of the transformer, followed by 3 2 60].
Another branch is improve the logical reasoning ability for abstract symbols, such learning the logical-based temples and expecting the model to generalize to unseen samples [10] 36]. These
studies closely related to languages, such as generating longer answering sequences solving mathematical problems in large language models, lacking explicit connections to visual tasks. [6] delves into the logical explanations in image classification by explicitly extracting logical relationships. While this logical-based approach sheds light on the interpretability of image classification models, its specific benefits for visual generalization remain relatively unexplored.
Multi-domain generalization. Current approaches for mDG in image classification focus on learning invariant representation across domains. Previous approaches like DANN [20] minimize feature
divergences between source domains. CDANN [35], CIDG [34], and MDA 23 consider conditions
for learning conditionally invariant features. MIRO 25 and GMDG (50) take advantage of pre-
trained models to improve generalization. Specifically, in comparison to MIRO, GMDG proposes general entropy- based learning objective for mDG and sufficiently minimizes the domain gaps, yielding better generalization results.
Generalized category discovery. Generalized category discovery, pioneered by [54] addresses unlabeled samples with both known and unknown classes. Furthermore, PIM [16] integrates InfoMax into generalized category discovery, effectively handling imbalanced datasets and surpassing GCD on both short- and long-tailed datasets.
Conclusion
This paper presents L-Reg, a logical regularization approach tailored for image classification tasks using logic analysis frameworks. L-Reg yields better generalization across different settings by fos- tering balanced feature distributions and streamlining the classification model's complexity. Rigorous theoretical analyses and empirical validations underscore its efficacy, as L-reg consistently improves generalization performance with different frameworks under various scenarios.
Limitation. L-Reg narrows the extent of semantic supports, potentially diminishing the amount of information available for classification and leading to certain trade-offs in the performance of seen datasets. This effect is evidenced by the slight decline in the accuracy of known classes when L-Reg is applied, as shown in Table A similar phenomenon is observed in Fig. [5] where the model fails to recognize person in the sketch domain lacking facial features. Analysis from Table f suggests that these compromises may result from improper 2. Future work should focus mitigating potential compromises on seen datasets by exploring strategies for better capturing Z through improved model architecture design. We offer more experimental results of possible solutions to this limitation in Appendix G such as further constraining the independence of each dimension in Z. Those results may suggest a direction for future work.
Acknowledgments
The work was partially supported by the following: National Natural Science Foundation of China under No. 92370119, No. 62376113,, No. 62206225, and No. 62276258.
10

---

References
[1] Emmanuel Abbe, Samy Bengio, Aryo Lotti, and Kevin Rizk. Generalization the unseen, logic reasoning and degree curriculum. In International Conference on Machine Learning, pages 31-60. PMLR, 2023.
[2] Emmanuel Abbe, Elisabetta Comacchia, and Aryo Lotfi. Provable advantage curriculum learning on parity targets with mixed inputs. Advances in Neural Information Processing Systems, 36, 2024.
[3] Kartik Ahuja and Amin Mansouri. On provable length and compositional generalization. arXiv preprint arXiv:2422.4887 2024.
[4] Hajnal Andréka, István Németi, and Ildikó Sain. Universal algebraic logic. Studies in Logic, Springer, due to, 2017.
[5] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.0289)) 2019.
[6] Pictro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, and Stefano Melacci. Entropy- based logic explanations of neural networks. In Proceedings of AAAI Conference on Artificial Intelligence, volume 36, pages 6046-600 000--055,222 2022.
[7] Sara Beery, Grant Van Hor, and Pietro Perona, Recognition terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456-473, 2018.
[8] Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research, 22(1):46-100, 2021.
[9] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks new unlabeled sample. Advances in neural information processing xystems, 24, 2011.
[10] Enric Boix-Adsera, Omid Saremi, Emmanuel Abbe, Samy Bengio, Etai Littwin, and Joshua Susskind. When can transformers reason with abstract symbols? arXiv preprint arXiv:2310.09755 2023.
[11] Malik Boudiaf, Imtiaz Ziko, Jérôme Rony, José Dolz, Pablo Piantanida, and Ismail Ben Ayed. Information maximization for few-shot learning. Advances in Neural Information Processing Systems, 33:2445-2457, 2020.
[12] Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specifie features to enhance domain generalization. Advances in Neural Information Processing Systems, 34:21189 29221101, 2021.
[13] Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In International Conference on Learning Representations, 2022.
[14] Mathilde Caron. Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[15] Zhuomin Chai, Yuxiang Zhao, Wei Liu, Yibo Lin, Runsheng Wang, and Ru Huang. Circuitnet: An open-source datasel for machine learning in vlsi cad applications with improved domain-specific evaluation metric and learning strategies. IEEE Transactions on Computer-Aided Design Integrated Circuits and Systems, (2))::303445 2023.
[16] Florent Chiaroni, Jose Dolz, Ziko Intiaz Masud, Amar Mitiche, and Ismail Ben Ayed. Parametric infor- mation maximization for generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1729-1739, 2023.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[18] Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pages 1657-1664, 2013.
[19] Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9284 9284-9292, 2021.

---

[20] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette. Mario Marchand, and Victor Lempitsky. aovaaiiadversaaiill training of neural networks. The journal of machine learning research, 17(1):096-20300, 2016.
[21] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01444 2020.
[22] Kai Han, Sylvestre-Alvis Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[23] Shouho Hu, Kun Zhang, 7hitang Chen, and I Laiwan Chan. Domain generalization via multidomain discriminant analysis. In Uncertainty in Artificial Intelligence, pages 292-302. PMI.R, 2020.
[24] Zeyi Huang. Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domane generalization. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28. 2020, Proceedings, Part 16, pages 124 140. Springer, 2020.
[25] Cha Junbum, Lee Kyungjae, Park Sungrae, and Chun Sanghyuk. Domain generalization by mutual- information regularization with pre-trained models. European Conference on Computer Vision (ECCV). 2022.
[26] Dachee Kim. Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9619-9628, 2021.
[27] Andreas Krause, Pietro Perona, and Ryan Gomes. Discriminative clustering by regularized information maximization. Advances neural information processing systems, 23, 2010.
[28] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization, In Proceedings of the IEFE international conference on computer vision workshops, pages 554-561, 2013.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009.
[30] David Krueger, Fthan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-ddistribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pages 5815-5826. PMLR, 2021.
[31] Da Li, Yongxin Yang. Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-Icarning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[32] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision, pages 5542-5550, 2017.
[33] Haoliang Li. Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400-5409, 2018.
[34] Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. conditional invariant representations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Domain generalization via
[35] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European conference on computer vision (ECCV), pages 624-639, 2018
[36] Zenan Li, Yunpeng Huang, Zhaoyu Li, Yuan Yao, Jingwei Xu, Taolue Chen, Xiaoxing Ma, and Jian La. Neuro- suuro-yymbolic learning yielding logical constraints. Advances in Neural Information Processing Systems, 36, 2024.
[37] Ziyue Li, Kan Ren, Xinyang Jiang, Yifei Shen, Haipeng Zhang, and Dongsheng Li. Simple: Specialized model-sample matching for domain generalization. In The Eleventh International Conference on Learning Representations, 2022.
[38] MacQueen. Classification and analysis of multivariate observations. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, pages 281-297, 1967.
12

---

[39] Ofir Moshe, Gil Fidel, Ron Bitton, and Asaf Shabtai. Improving interprelability via regularization of neural activation sensitivity. arXiv preprint wrXiv:2211.08655 2022.
[40] Hyconseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In Proceedings of the IEFF/CVF Conference Computer Vision and Pottern Recognition, pages 8690-8699, 2021.
[41] Konstantinos Panagiotis Panousis, Dino lenco, and Diego Marcos. Sparse linear concept discovery models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2767-2771, 2023,
[42] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2019.
[43] Wilhelm Rödder. Conditional logic and the principle of entropy. Artificial Intelligence. 117(1):83-106 2000.
[44] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint 10t.::99110073.11 2019.
[45] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference computer vision, pages 618 626, 2017.
[46] Yuge Shi, Jeffrey Seely, Philip Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.099) 2021.
[47] Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateck Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 804-814, 2022.
[48] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCC ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 443 450. Springer, 2016.
[49] Kiat Chuan Tan, Yulong Liu, Barbara Ambrose, Melissa Tulig, and Serge Belongie. The herbarium challenge 2019 dataset. arXiv preprint 1rXiv:1906.05372 2019.
[50] Zhaorui Tan, Yang, and Kaizhu Huang. Rethinking multi-domain generalization with general learning objective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23512 -33522, June 2024.
[51] Zhaorui Tan, Xi Yang, and Kaizhu Huang. Semantic-aware data augmentation for texl-to-image synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5098-5107, 2024.
[52] Irene Tsapara and György Turán. Learning atomic formulas with preseribed properties. In Proceedings of the eleventh annual conference on Computational learning theory, pages 166-174, 1998.
[53] Vladimir N. Vapnik. Statistical Learning Theory. Wiley- Interscennc. 1998.
[54] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7492-7501, 2022.
[55] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5018-5027, 2017.
[56] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-ucsd birds-200-2011 dataset. 2011.
[57] Chunyang Wu. Mark JF Gales, Anton Ragni, Penny Karanasou, and Khe Chai Sim. Improving inter- pretability and regularization in deep learning. IREF/ACM Transactions on Audio, Speech, and Language Processing, 26(2):256-265, 2017.
[58] Mike Wu, Michael Hughes, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi Velez. Beyond sparsity: Tree regularization of deep models for interpretability, In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
13

---

[59] Mike Wu, Sonali Parbhoo, Michael Hughes, Ryan Kindle, Leo Celi. Maurizio Zazzi, Volker Roth, and Finale Doshi-Velez. Regional tree regularization for interpretability in deep neural networks. Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6413-6421, 2020.
[60] Changnan Xiao and Bing Liu. A theory for length generalization in learning reason. arXiv preprint arXiv:2404. 2024.
[61] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning adapt to domain shift. Advances Neural Information Processing Systems, $4:23664-23678, 2021.
[62] Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint arXi::2104.0000 2021.
[63] Jialy Zou, Xinggang Wang, Jiahao Guo, Wenyu Liu, Qian Zhang, and Chang Huang. Circuit as points, Advances in Neural Information Processing Systems, 36, 2024.
14

---

A Broader impact
Our regularization term based on logic for image classification offers significant potential beyond academia. By integrating logical constraints, our approach enhances model robustness, interpretability. and ethical alignment. This translates into improved performance on real-world tasks such as disease diagnosis in healthcare and mitigating biases in decision-making systems. Our work fosters interdisciplinary collaboration and contributes the responsible deployment of Al technologies, ultimately benefiting society through enhanced efficiency, fairness, and transparency in machine learning applications.
B Details of the logical framework for visual classification task
We provide more details of the connections between logical reasoning and visual classification tasks. Definition B.1. Following 4 a logic L a five-tuple defined in the form:
L = (Fc,
(16)
where
Fz is a set of all formulas of C. Fc arbitrarily refers to any collections that can be 'expressed' by language L. Therefore, F& could be not only collection of languages but also images and labels (X, Y) for computer vision cases.
Mc is class called the class of all models (or possible worlds) of C; intuitively, this can be considered as different domains D of X.
Fc is binary relation, FcS Mc Fz, called the validity relation of L. For example, in the known set, the ground truth label of the image is given as truth, which the validity relation.
mnge: FE X Mc - Sets where Sets is the class of all sets. mnge is a function with domain Fc. x Mc, called the meaning function of C. Intuitively, magg extracts the meaning of the expressions that can be understood as the classifiers.
- represents the provability relation of L, telling us which formulas are 'true' in which possible world and usually is definable from mngg, such as the estimation criteria in the machine learning system.
Accordingly and still following good general logic is defined as:
Definition B.2 (General logic). A general logic is class:
L- === P Sigig.
(17)
where Sig is a class of sets; = = DE.:Hemmgg'' mnng,,
-P. - > is a compositional logic in the sense of Definition B.I for P E Sig, and for any sets P. E Sig satisfies the following conditions:
the set of atomic formulas of LP.
2. Cn(CP) = Cn(L9)= == Cn((*) where On(-) is called the set of logical connectives the given logic (these are operation symbols with finite or infinite ranks).
3. Any bijection f: PBQ that extends to a bijection between the tautological formula algebras of L and _ induces an isomorphism between L and L2
4. If P Q, then L is sublogic of L
5. For any P E Sig and set H, there is a Pt e Sig such that P' is disjoint from H and L is isomorphic copy of L..
6. The union of system p, I of pairwise disjoint sets P. from Sig belongs to Sig. whenever is not empty. Let Br(-) denotes free algebra, Alg,, represents {mngm (3) M .M} where denotes the term algebra. Further, the tautological congru- ence of the logic belonging to the disjoint union P is generated in St (Alg.. (CP). P) as
15

---

congruence by the union of the tautological congruence relations of the logics belonging to P., i I.
7. Sig contains at least one non-empty set.
Our L-Reg aims to regularize the semantics extracted by and the classifier to satisfy condition
in Eq. (6) can be safely omitted in the rest of the paper. Consider the logic formed on X,Y: (x),))) (F), D, (x, he, (((x)))))- Assume want study the logic of I- which be defined in the form of L+ del (Fx,, Y., Dr, 1,,4,,-- are pseudo-components associated with H. Particularly, D- is a subset of all possible world/domains from F(x.,y.) Dr def {T S F(x.,))
where
is closed under F(N(x),r)) For any ebe F F (a + 6)}. Further, F- in T D2 is defined as T Fー < T. 4 points out that the following condition is almost always satisfied: (Cond) Va. b E F-, d D.., we have (h- d) h (b, d)) and Fi = F Therefore, the semantical consequence relation induced by F+ coincides with the original syntactical F (hog(X)).)) while Cond holds. Due to that Dr S D, it infers that (g((X)),X,) coincides with F+. Therefore, (((()))))))) can be safely omitted in the rest the paper.
T DD and E Y.), it has (a, del
C Details of proofs
Proposition C.1 (L-Reg reduces the complexity of the model, promoting data-shift generalization performance.). Assume the domain gap is well minimized. Consider a 1 is the target model that generalizes to the unseen with the lowest complexity. There are ( _(xx..,
((x,, trained under the
setting of data-shift generalization (i.e., (X Y,) is accessible and y, Val, it has that:
GIA X.) S GL(J(X.YY); N.),,
(18)
Proof. We assume the loss is achieved for the tractable form by minimizing the mean squared error. In that case, we have (x..Y.) for the given training set as:
(19)
In comparison to fr, f(X..Y)) for the given seen sets is as:
f(x,x)) = 2z,
(20)
and 1 l(x.Y..) is derived from f(x,Y,), where 7, is constrained additionally by L-Reg and the constrained Z, denoted as Z
fBx,,.)) = (Z RT 2.) -,, yY,,
(21)
For simplification, denote (Z (2, - and (Z) N and AR respectively.
The form of N. For multi-domain generalization, the model is tested on the unseen domain, referring that X2, contains some unseen semantics besides the seen: Z, Z,, Zu Zg. Zg + Zu, Z, Z,, * Ø. Considering each dimension of Z represents specific semantics. we denote I as the dimensions of Z that contain the seen semantics support X, and r for the unseen, we can decompose N as:
rTr FT FTF
-1 [ACT) A(D))"
== =
(22)
The form * Assume already contains semantic support for deducting Y; thus, T would not affect the deduction of Y. In such case, it has that IT 0 and ITr 0 and IT where 0, 1
16

---

denote zero matrix and identity matrix:
T 0
N
(D(T) A(T)??
(T
(h(T) h(T))"
(23)
(((TT)->(L)))
h(T')
where we also expect h(T) = 0 so that zu does not influence the deduction. We now have N
r
N
(&T) A(T)), A.L,A(T) 0.
(24)
Note that for N in f(x,,))) ITT and I'T are not constrained. Please refer to Lemma C.2 more, hT) is also not constrained.
Further-
The form of R Now we discuss the trainable AR obtained with the application of L- Reg. The form of N is similar to V However, Eq. (11) indicates that L-Reg minimizes | T 2 and [|[*T||2 through- (Y (Y))))))) D) and also minimizing [/([T)]2:
rTT FTT
NR
(A(T) A(T)), s.t. minn|I/TT || + [T(T))
(25)
TTT
Compare GL(F (X),)) f" Xu) with GL(f(x,,))) Xu). By comparing the forms of N ,N and No, obvious that IWR N S || N Therefore, we have that: GLU (X..)) X.) < GT( f", fu).
n
Lemma C.2 (Minimizing HY (X))))) D) R solely may cause generalization degradation). Mini- mizing HY ,,((), D) + R solely without L-Reg may conflict with maxh,9 a (I'),"), D), causing invalid semantics for decision process and degrading the generalization.
Proof. We have the following relationship for H(Y [g(2), D):
H(Y]g(z), D) = H1 ( g(I),, g(T), H(Y, (TV.g(T))((T))))))))))))))))())) D) H()(T))a(T)) D) =
(26)
gT)) D) H(((T))((T)) D).
Since the independence between {z J-M is unconstrained, g D) may cause that Y be deducted from r. Therefore, I and ITT are not constrained even when the domain gap is minimized where [D] = 1, causing the sub-optimal generalization.
D One toy example
We present a simplified informal illustrative example to compare the efficacy of our proposed L-Reg against conventional [.1 and 1.2 regularization methods. As depicted in Fig. 6 the ground truth (GT) image represents the underlying data, generated according to f" ("1,, x2) = sin(2rr.1) (2 where æ and T2 denote the horizontal and vertical coordinates respectively, and the pixel color corresponds to the value of (I1,, 22). The training domain is delineated by the black box, while the testing domain encompasses the area outside of this boundary.
For our experiments, we use 6-linear-layer size- 110 ReLU model network. Mean squared error serves the loss function.
Our experimental results reveal that L-Reg enhances the model's ability extrapolate beyond the training domain. Notably, our proposed L-Reg demonstrates superior extrapolative capabilities compared to traditional Ta and T-2 regularization methods. This observation highlights the efficacy of L-Reg in fostering improved generalization.
17

---

GT
Base model
1 Regularization La Regularization
L-Reg
Figure 6: Prediction visualizations MLP with different regularization terms.
E
Apply L-Reg to ERM Baseline for mDG
To further validate L-Reg's efficacy for mDG, we use ERM as the baseline on the Terralncognita dataset. For fair comparison, all experiments share the same hyperparameter settings and use the Regnety- 16gf backbone. Original ERM results are also included alongside our reproduced results. The results Table reveal that ERM with L-Reg significantly improves mDG performance (from 49.9% to 52.9%).
F Compare L-Reg with more regularization terms
We also compare L-Reg with other regularization terms: ization that constrains the independence of each dimension of the semantic feature 2; and Sparsity implemented as Bernoulli Sample of the latent features from the sparse linear concept discovery models [41] our used PIM backbone. To investigate this fairly, re-implemented the Bernoulli Sample of the latent features from the Sparse Linear Concept Discovery Models [41] on the same PIM backbone that used, to achieve the sparsity. Table TTale demonstrate that L-Reg outperforms Ortho-Red Reg and Sparsity.
The Ortho-Reg the orthogonality regular-
Especially, while common sparse concept model may be able to achieve 7/yh 2 by filtering irrelevant features through the sparsity, may not ensure 7 yV, which is crucial for disentangling features used for predicting different classes. This limitation can potentially lead to degradation in generalization performance for common sparse concept models. 6 indicate that while L-Reg consistently achieves overall improvement, the sparse concept-based approach does not consistently improve generalization, validating the aforementioned difference.
G Limitation of L-Reg and possible solutions
As analyzed and discussed in the paper, L-Reg is based the precondition that each dimension of the latent features represents an independent semantic.
We hypothesize this is due to the fact that L-Reg is derived based on the precondition that z', 2²,²' x E 2, 1 전 j is independent each other. This condition holds for most deep-layer features but may not apply to shallow layers. Thus, applying L-Reg to the semantic features from the deep layers may improve the performance for unknown classes without negatively impacting known classes.
Derived from this hypothesis, another possible solution is further regularizing the independence, which may lead to further improvements. validate this hypothesis, we test L-Reg by reinforeing independence with Ortho-Reg. MDG results in Table GCD results in T& combining L.-Reg with Ortho-Reg leads to further improvements, whereas Ortho-Reg alone may not guarantee improvements. These findings support hypothesis and suggest that L-Reg, particularly
show that
18

---

Table Results of GCD: Averaged results across all datasets of PIM with different regularization applied to the latent features: Sparsity: achieved through Bemoulli Sample; Ortho-Reg: orthogonality regularization. +L--Reg outperforms other regularization terms when they are applied solely; +I.- Reg+Ortho-Reg achieves the best performance and alleviates the performance degradation of unknown classes, validating our hypothesis in the paper that the improper may result compromises and constraining the independence of each 2' E Z may be helpful.
Avg
All
Known
Unknown
PIM
67.4
79.3
59.9
+Sparsity Improvements
66.6
77.3
60.0
-2.0
+Orhh-RRg
68.4
79.2
61.9
Improvements
1.0
-0.1
2.0
+L-Reg
68.8
79.0
62.7
Improvements
1.4
-0.3
69.3
79.6
63.4
Improvements
2.0
0.3
3,5
Table 7: Results of GCD: Detailed results across all datasets of PIM with different regularization
applied the latent features: Sparsity: achieved through Bernoulli Sample; Ortho-Reg: orthogonality regularization.
CUB
Stanford Cars
Herbarium 19
All
Known
Unknown
All
Known
Unknown
All
Known
Unknown
PIM
62.7
75.7
56.2
43.1
66.9
31.6
42.3
56.1
34.8
PIM + Sparsity
60.1
72.7
53.8
40.4
61.7
30.1
42.0
53.7
35.8
Improvements
-2.6
-3.0
-2.4
-2.7
-5.2
-1.5
-0.3
-2.4
1.0
PIM + Ortho-Reg
64,9
76.7
58.9
44.3
65.6
34.1
42.9
57.2
35.1
Improvements
2.2
1.0
2.7
1.2
-1.3
2.5
PIM L-Reg
65.3
76.0
60.0
44.8
66.0
34.6
43.7
55.8
37.2
Improvements
2.6
0.3
3.8
1.7
-0.9
3.0
-0.3
2.4
PIM L.Reg Ortho-Reg
66.8
77.3
61.6
45.8
67.3
35.5
43.3
57.5
35.6
Improvements
4.1
1.6
5.4
2.7
0.4
3.9
1.0
1.4
0.8
CIFARIO
CIFARI00
ImageNet PeaggeNet-ION
All
Known
Unknown
All
Known
Unknown
Known Unknown
PIM
94.7
97.4
93.3
78.3
84.2
66.5
83.1
95.3
77.0
PIM + Sparsity
94.2
97.4
92.6
79.7
84,6
69.7
93.7
78.2
Improvements
-0.5
0.0
-0.7
1.4
0.4
3.2
-1.6
1.2
PIM Ortho-Reg
95.1
97.4
93.9
80.2
84.6
71.4
83.0
93.4
77.7
Improvements
0.4
0.0
0.6
1.9
0.4
4.9
-0.1
-1.9
0.7
P
94.8
97.6
93.4
80.8
84,6
73.2
83.4
94.0
78.0
Improvements
0.1
0.2
0.1
2.5
0.4
6.7
0.3
-1.3
1.0
PIM L-Reg Ortho-Reg
95.1
97.6
93.9
81.2
84.2
75.0
83.7
93.6
78.7
Improvements
0.4
0.2
0.6
2.9
0.0
8.5
0.6
-1.7
1.7
when applied to deep layers in conjunction with Ortho-Reg, is beneficial. This suggests direction for future work.
H More experimental details and results
All experiments can be conducted one NVIDIA GeForee RTX 3090 GPU,
H.1 Multi-domain generalization
Competitors. We listed results from previous important work in the mDG field for better validation.
They are: MMD 33). Mixstyle [62) GroupDRO [44] IRM [5), ARM (61) VREx 30 CDANN (5)
DANN 20 RSC 224, MTL g. MLDG BI, Fish [46). ERM (3), SagNet [40]. SelfReg [26].
CORAL 48 mDSDI 12], MIRO (55, and GMDG [50]. Among them, GMDG is treated as our
baseline since it sufficiently minimizes the domain gaps.
Datasets. We use PACS (4 domains, 9,991 samples, classes) 32
VLCS (4 domains, 10, 729
samples, 5 classes) [8]. OfficeHome (4 domains, 15,588 samples, 65 classes) [55), TerraIncognita
19

---

Table 8: Results of mDG: Results of using ERM as the baseline. We use the ERM method as the baseline to test L-Reg's efficacy. Ortho-Rege Reg: orthogonality regularization. This table includes results: (1) The improved performance of L-Reeg on ERM baseline. (2) Comparison between L-Reg and the Ortho-Reg on ERM baseline. (3) Using I.-Reg and Ortho-Reg together yields further promotion, validating our 'improper z' hypothesis in the Paper limitation part. The used dataset is Terralncognita. All experiments share the same hyperparameters except the added regularization term. Each group of experiments is run with seeds [0,1.2], and the averaged results for each domain and additionally with the standard deviation (Std) are reported.
Terralncognita
Location 100
Location 38
Location 43
Location 46
Avg + Std.
ERM
54.3
42.5
55.6
38.8
47.8
ERM Reproduced
50.6
49.7
58
41.2
49.9+3.6
+Ortho-Reg
50.7
52.6
60.5
42.7
51.6+2.5
+eg
52.7
51.7
61.3
45.8
52.9+4.2
+L-Reg+Ortho-Reg
61.5
48.6
60.3
44
53.620.5
Table 9: Parameters for mDG task
Use RegNetY-16G0 16GF
Ir mult
a
Terralncognita
2.5
le-3
OfficeHome
0.1
le-3
VLCS
0.1
lc-4
PACS
0.1
5e-4
DomainNet
5.0
1e-3
(Terralncognita, 4 domains, 24, 778 samples, 10 classes) and DomainNet (6 domains, 586,575
samples, 345 classes) 422
Training details. We use GMDG 50 as our baseline. Especially, we use all loss terms proposed in GMDG as Lmain The training procedure the same as MIRÓ 25 and GMDG. We use seeds 0, 2,2 for all three trails training.
Parameters. We adhere to the parameters proposed by GMDG, particularly focusing its recom- mended loss terms. Furthermore, we provide detailed listing of the hyper-parameters pertaining to L-Reg, along with the tuned 'Ir mult', as outlined in Table facilitate the reproducibility of our results.
Evaluation metric. The models undergo training known domains and subsequent testing unseen domains. For each trial, a distinct domain within the datasets is designated as the unseen domain. The evaluation metric reports the prediction accuracy achieved on these unseen domains. The aggregated results across all unseen domains within the datasets provide a comprehensive assessment of the algorithm's performance in domain generalization for the given datasets.
More results. Results of each domain for each dataset are presented in Tables 10 to
H.2 Generalized category discovery
Competitors. We compare our proposed method with existing generalized category discovery methods: GCD (54) and PIM [6]. In particular, PIM based on information maximization the current state-of-the-an (SOTA) generalized category discovery method. Additionally, the traditional
machine learning method, k-means [38]; three novel category discovery methods: RankStats+ [22]. UNO+ [191. ORCA [13;; and several information maximization methods: RIM (27]. and TIM [I] are adapted for generalized category discovery as competitors. The results of the modified novel category discovery methods are reported in [54], and the modified information maximization methods
are reported in [6]]
Usage details of datasets for GCD. Following the protocols of GCD and PIM [4 [6], the initial training set each dataset divided into labeled and unlabeled subsets; samples from half of the classes are assigned as unlabeled, and their labels are not used for training. Specifically, half of the image samples from known classes are allocated to the labeled subset, while the remaining half are assigned to the unlabeled subset. Additionally, the unlabeled subset includes all image samples from
20

---

the novel classes in the original dataset, As result, the unlabeled subset consists of instances from K different classes. The detailed statisties of datasets are listed in Table
Training details. Consistent with PIM, we utilize latent features extracted by the feature encoder DINO (VIT-B/16) [14] that is pre-trained on ImageNet [77] through self-supervised learning. The losses proposed in PIM are treated as Lmain. The original PIM freezes the feature extractor during the training, directly using the pre-saved extracted features as the model input. For fair comparison, we only added one linear layer as g on the extracted features, which is the minimal modification.
L2 (weight decay) value searching. For more fair comparison, we conduct weight decay value searching to ensure that the weight of Lg is the best. To address this, we devised a methodology for weight decay searching involving the construction of smaller labeled and unlabeled subsets derived solely from the labeled data. To conduct parameter searching, we split the labeled samples to construct 'smaller' sub-labeled and sub-unlabeled set. Specifically, take 50% of the samples from known classes as sub-unlabeled samples from unknown classes. Additionally, we take 25% of the samples from the remaining 50% of known classes sub-unlabeled samples from known classes. The remaining samples are treated as sub-labeled samples, Hyper-parameters are then searched on these sub-labeled and sub-unlabeled sets.
Parameters of L-Reg. The hyper-parameters of L-Reg values are shown in Table 16
Evaluation metric. Following prior works S4 16 we use the proposed accuracy metric from 54 of all classes, known classes, and unknown classes for evaluation.
More results. The results for each dataset are presented in Table 17 It is evident that L.-Reg yields enhanced performance across half of the datasets for both known and unknown classes. On the remaining datasets, while L-Reg may slightly compromise the performance of known classes, it demonstrates significant improvements in the unknown classes, resulting in an overall enhancement in the performance across all classes.
More ablation results. Due to the introduction of tuned weight decay and the additional component, we have conducted ablation studies to assess their impact. The results are summarized in Table [8] is observed that the baseline model utilizing the tuned weight decays performs slightly better than the original weight decay settings. Notably, the tuned weight decays contribute to improvements in unknown classes while often leading slight decreases in known classes across most datasets. Inclusion of the proposed extra component g results in marginal improvements in both known and unknown classes compared to the tuned baseline. Our proposed L-Reg demonstrates significant improvements specifically in the unknown classes, thereby corroborating Proposition 4.2 However, as discussed in the main paper, it is acknowledged that L-Reg may entail compromises in the performance of known classes.
H.3 Combination of multi-domain generalization and generalized category discovery
Datasets. We leverage the datasets utilized in mDG tasks to construct the mDG+GCD datasets. Specifically, during the seen domains of training, labels from approximately half of the classes are masked. For instance, in the PCAS dataset comprising 7 classes, classes labeled within the range [0, 1, are retained, while classes in [4, 5, 6) are masked. as unknown classes in our setup are from unknown classes. However, we acknowledge that this prior is not explicitly known. To align with the GCD setting, we operate under the assumption that the unlabeled set may potentially include samples from known classes. Consequently, we refrain from constraining the model by mandating that unlabeled data be classified solely as unknown classes. This adjustment introduces more challenging generalization scenario.
noteworthy that data categorized
Training details. For all experiments, the implementation directly adds L-Reg to their previously proposed loss sets. The models are trained with the aforementioned labeled and unlabeled sets from the seen domains and tested on the samples from the unseen domain.
Parameters. We include all the parameters for reproducing our experiments in the code. Please refer to the code for details.
Evaluation metric. We use the same metric from the GCD task for the mDG+GCD task, Similarly, the metrics include the accuracy for all, known and unknown classes.
21

---

More results. The averaged results of each dataset are exhibited in Table 19 while the detailed results of each dataset are presented in Tables 20 to 24
H.4 More GradCAM visualizations
We provide more visualized examples of L-Reg. Examples of known classes can be seen in Figs. 7
to T0 and unknown classes in Figs. and 12 limitations, can be seen in Figs. and
Compromises in known sets, discussed in the
22

---

Table 10: MDG experiments on Terralncognita: More results of full GMDG+L-Reg for each category.
TerraIncognita
Location 100
Location 38
Location 43
Location 46
Avg.
ERM
21
54.3
42.5
$5.6
38.8
47.8
MIRO
25
(usc ResNet-50)
50.4
GMDG
50
(usc ResNet-50)
59.8=1.0
45.3+1.7
57.1+1.8
38,2+5
50.1+1.2
MIRO
25
(use RegNecY-16GF)
58.941.3
GMDG
50
(use RegNetY-16GFF
73.3+3.3
54.7+1.4
67.1=0.3
48.626.5
60.7+1.8
GMDG + Reg (use RegNecY-16GF))
73.9=0.8
57.1+2.3
67.9#1.1
52.7=4.0
62.9+0.9
Table 11: MDG experiments on OfficeHome: More results of full GMDG+L-RRg for each category.
OfficeHome
art
clipart
product
real
Avg.
ERM
21
63.1
51.9
77.2
78.1
67.6
MIRO
25
(use ResNet-50)
70.5+0.4
GMDG
50
(use ResNet-50)
68.940.3
56.2+1.7
79.9+0.6
82.0+0.4
70.7+0.2
MIRO
25
(use RegNetY-16GF)
80.4+0.2
GMDG 50
(use RegNetY-16GF)
79.7+1.6
67.7+1.8
87.8+0.8
87.920.7
80.8+0.6
GMDG I.-Reg (use RegNerY-16GF)
78.420.3
69.3+0.7
87.920.6
88.0#0.8
80.9
Table 12: MDG experiments on VLCS: More results of full GMDG+L-Reg for each category.
VLCS
caltech101
labelme
sun09
voc2007
Avg.
ERM
21
97.7
64.3
73.4
74.6
77.3
MIRO
25
(use ResNet-50)
79.0+0.0
GMDG
50
(use ResNet-50)
98.3+0.4
65.9+1
73,4+0.8
1.3
79.2+0.3
MIRO
2
usc RegNetY-16GF)
79.9=0.6
GMDG 50
(use RegNetY-16GFF)
97.9+1.3
66.8+2.1
80.8#
83.9 1.8
82.440.6
GMDG L-Reg RegNetY-16GF)
98.6+0.1
67.1#0.1
80.7+0.7
83.0#0.8
82.4+0.0
Table 13: MDG experiments on PACS: More results of full GMDG+L-Reg for each category.
PACS
art paintinn
cartoon
photo
sketch
Avg.
ERM 21
84.7
80.8
97.2
79.3
84.2
MIRO 25
(use ResNect-50)
85. 85.4+0.4 4
GMDG 50
(use ResNet-50)
84.7+1.0
81.7+2.4
97.5+0.4
80.5+1.8
85.6#0.3
MIRO
25
(use RegNetY-16GF)
97. 97.420.2
GMDG
50
(use RegNetY-16GF)
97.5
97.0=0.2
99.4+0.2
95.2+0.4
97.3+0.1
GMDG + L-Reg (use RegNetY-16GH)
97.620.8
97.1+0.3
99.3+0.2
95.3+0.9
97.4+0.2
Table 14: MDG experiments DomainNet: More results of full GMDG+L-Reg for each category.
DomainNet
clipart
info
painting
quickdraw
sketch
Avg.
21
50.1
21.2
63.7
13.9
52.9
44.0
MIRO 25
(use ResNet-50)
44.3+0.2
GMDG
(use ResNet-50)
63.4+0.3
22.4+0.4
51.4204
13.4+0.8
64.4+0.3
52.4+0.4
44.6+0.1
MIRO 25
(use RegNetY-16GPP
53.8+0.1
GMDG
(use RegNetY-16GI)
74.0+0.3
19.5
61.540.3
16.3+1.2
73.9+1.5
62.8+2.4
54.6+0.1
GMDG + L-Reg (use RegNetY-16GF)
74.1#0.1
42.6+1.0
62.3+2.9
12,740.9
75.920.8
64.620.2
55.4:0.0
23

---

Table 15: Statistics of datasets.
CUB
Standford Cars
Herbarium19
CIFARIO
CIFARI00
ImageNet- 100
Known classes
100
98
341
5
80
50
Seen data
1.5K
2.0K
8.9K
12.5K
20K
31.9K
All classes
200
196
683
10
100
100
Unseen data
4.5K
6.1K
25.4K
37.5K
30K
95.3K
Table 16:
GCD task: Tuned weight decay values for each dataset,
CUB
Standford Cars
Herbarium19
CIFARIO
CIFAR100
ImageNet-100
Tuned weighted decay a of L-Reg
0.0272
0.02/2
0,0272
0.0572
0.00572
0.00572
0.1
0.001
0.2
0.01
0.0025
0.01
Table 17: GCD results: Accuracy scores across fine-grained and generic PIM datasets with our L-Reg and other competitors. The best results of each group are highlighted in bold. Improvement and degradation in our approach from PIM are highlighted in red and blue, respectively.
Stanford Cars
Herharium
Approach
All
Known
Unknown
All
Known
Unknown
All
Known
Unknown
K K-means
34.3
38.9
32.1
12.8
10.6
13.8
12.9
12.9
12.8
RankStats+
(22] (TPAMI-21)
33.3
51.6
24.2
28.3
61.8
12.1
27.9
55.8
12.8
UNO+
19
(ICCV-21)
35.1
49.0
28.1
35.5
70.5
18.6
28.3
53.7
14.7
ORCA
(ICLR-22)
27.5
20.1
31.1
15.9
17.1
15.3
22.9
25.9
ORCA
13
VITB16
38.0
45.6
31.8
33.8
$2.5
25.1
25.0
30.6
19.8
GCD
54
(CVPR-22)
51.3
56.6
48.7
39.0
57.6
29,9
35.4
51.0
27.0
InfoMax hased methods
RIM (NcurIPS-10)
52.3
51.8
52.5
38.9
57.3
30.1
40.1
TIM
11
(NcurIPS-20)
53,4
51.8
54.2
39.3
56.8
30.8
40.1
57.4
30.7
PIM
16
(ICCV223)
62.7
75.7
56.2
43.1
66.9
31.6
42.3
56.1
34.8
PIM
L-Reg (Ours)
65.3 S1
76.0 9:3
60.0
44,8 71
66.0
34
43,72
85.80aa
37.2
CIFAR10
CIFAR100
Image Net-100
Approach
All
Known
Unknown
All
Known
Unknown
All
Known
Unknown
K-means
83.6
85.7
82.5
52.0
52.2
50.8
72.7
75.5
71.3
RankStatst
22
(TPAMI 21)
46.8
19.2
60.5
58.2
77.6
19.3
37.1
61.6
24.8
UNO+
(CCV-211)
68.6
98.3
53.8
69.5
80.6
47.2
70.3
95.0
57.9
ORCA
13
(ICLR-23
88.9
88.2
89.2
55.1
65.5
34.4
67.6
90.9
$6.0
ORCA
13
VITB16
97.1
96.2
97.6
69.6
76.4
56.1
76.5
92.2
68.9
GCD
54
(CVPR
91.5
97.9
88.2
70.8
77.6
$7.0
74.1
89.8
66.3
InioMax based methods
RIM
(NeurIPS I0)
98.1
78.9
74.4
TIM
|
(NeurIPS-22)
93.1
98.0
90.6
73.4
78.3
63.4
76.7
93.1
68.4
(ICCV-23)
94.7
97.4
93.3
78.3
84.2
06.5
83.1
95.3
PIM
L-ure((Urr))
94.8" 11
97.6
93.4
80.8 81
84.6
73.2
83.4
94.0"-a1
78.0
Table 18: GCD results: Accuracy
across fine-grained and generic datasets of each setting. The
best results are highlighted in bold. To eliminate the impact of apper-prraeeerss on performance, we also present the results of PIM with tuned parammaarssssess (termed bascline tuned). Limai denotes the losses used in PIM. denotes the transformation applied to the input features.
CUB
Stanford Cars
Herburium19
ID
Settings
All
Known
Unknown
All
Known
Unknown
All
Known
Unknown
Baseline (Lmaar))
62.7
75.7
56.2
43.1
66.9
31.6
42.3
56.
34,8
Baseline tuned (lomcin)
64.8
75.1
59.6
42.6
59.3
34.6
43.1
52.0
35.4
Lmaam
57.4
Ours (Lemiin the Rey)
65.3
76.0
60.0
44.8
66.0
34.6
43.7
55.8
37.2
CIFARIO Known
CIFARIOU
ImugeNNtt
1D Settings
All
Unknown
All
Known
Unknown
All
Known
Unknown
Basellin (Lematt)
77.0
Bascline tuncd (Lousin)
95.0
96.1
94.4
80.3
84.6
71.8
83.5
95.0
77.7
94.7
97.5
93.3
80.8
84.6
73.1
83.1
95.0
77.1
Ours 1g Rey
24

---

Table 19: MDG+GCD results: accuracy scores of each dataset. Improvements are highlighted in red.
HomeOftee
Terralnoggaio
Domotntet Unkmown
Mathod
Damsin
Unknow
Kannnn
Unkaswn
FRM
Net
52.36
77 23
20.31
44.50
74.67
8.50
51.51
82.59
34.88
17:34
20.46
45.15
22.56
40.80
6.85
+L-Reg
minimised
$5.86
77 mm
19:06
43.56
71.76
0.68
61.49
81.35
36.65
40.23
20.27
15.55
25.86
47.07
7.19
Inpooveneens
1.40
0.08
3.27
1:224
2.89
1.18
1002
11.55
1:77
3.36
18:01
19.58
3.31
6:15
0.34
71.05
43.42
72.44
65.19
90.34
40.24
42.99
7,86
1.21
0.77
3.43
MI 80 LL-Rex
Vinimined
56.85
65.02
24.85
48.3 73
30161
9:01
16.53
87-72
15.00
50.23
19:00 02
49.05
31.49
55.04
10:57
68.44
97.77
25,64
53,59
79.50
22:21
62.07
83.15
35.21
44.85
40.87
38.42
51.58
54.97
10.95
Imprreerrzie
11:61
12:14
0.99
9:91
-1.11
13:18
0146
0.15
-5.97
-11.0
0.10
-0:47
0:41
LL.Rex
misimized
86
31.33
82.77
36.09
38.75
55.18
11.30
9:90
0:40
21.15
X01
K99
40:54
2.54
7.99
0..0
0.83
Table 20: MDG+GCD results: accuracy scores of each domain in PACS dataset.
PACS Medhod
Arg
art painting
cartoon Known
photo
sketch
All
Kauwa
Unkaown
AllI
Known
Uskaown
All
Unknown
All
Knowa
Unlaows
All
Known
Cakaown
FRM
57.25
72.77
23:35
47.77
11:00
56.08
83.47
331.77
50:13
47:35
65.85
65.00
540:23
0130
with our my
55.86
77.69
19:05
45.33
85. 00.0
11.00
50.91
90.09
0.00
63:70
48.51
76.23
63:52
$6.75
01.00
Ingrevenents
-1.41)
-0.05
-3.27
-2.44
-4.60
(1.00)
-5.17
6.60
-20.47
4.57
1.16
7.38
-2.54
-3.48
11.00
PIM
56.35
71.06
27.43
46.30
55.17
57.32
50.37
89:15
0.00
62.05
49:50
72:40
66 19
50:40
0.00
with DET ma
58.47
76-49
26.22
46-74
83.05
0.00
56.50
78.77
20:57
64.90
48.51
77.33
65:35
50.62
0.00
Ingresements
2.12
5.43
-1.21
-0.06
32.87
-57.32
6.13
-10.38
27:57
2.25
-0.99
4.92
0.36
0.22
0.00
MIRO
56.85
$5.62
24:85
31.36
97.70
0.30
56.35
09.01
0.00
15.35
75.17
25.21
70.64
89:72
73.16
with OUE reg
68.46
97.97
25.64
68.46
97.82
35.24
61.51
98.02
14,09
72,60
98.84
50.96
71.18
96.39
2.26
Improwements
11.61
12:14
0.79
16.60
0.11
35.24
5.06
-1 1.89
14.00
24.25
23.68
24.75
0.54
26.87
-70.90
GMDG
38.35
91.46
10.15
51.92
97.82
0.30
54,90
96.95
0.00
56,14
74.85
40.73
70.45
96.22
0.20
with ONE reg
07.82
91.36
51.35
79.26
98,05
58.00
68.18
99.25
27.82
52.00
74.50
34.15
71.47
95.66
5.34
Improvements
9:50
0.40
21.15
27.33
0.23
38.00
13.38
2:26
27.82
3.24
0.33
6.36
1.02
0.56
5,34
Table 21: MDG+GCD results: accuracy scores of each domain in HomeOffice dataset.
HomeOffice Method
Ang
Art
Clipurt
Product
Real World
All
Known
Unkaown
All
Known
Unknown
All
Known
Unknnwn
Known
Uniaowe
All
Known
Unkoown
ERM
44.30
74.01
8:31
45:26
72.05
3:20
17.54
65..4
10.19
-5.71
15.74
9:87
49:28
$2.30
with ouE rog
43.56
71.7%
9.68
41.30
62.72
8.47
35.91
60.75
9.90
45.34
78.95
13.14
48.68
54.67
7.32
Improvementss
1.24
2.89
1.18
3,90
.996
5:22
2.03
3.710
0.29
1.63
0.21
3.27
0.60
1.85
3.46
PIM
43.22
72.84
8.13
42333
03.09
3.39
33.77
36.75
13.83
47.27
77.38
12:1
4K.11
57.33
2.50
with our ma
44.20
71.75
10:85
44.64
68.85
7.56
35.48
60.90
5.90
47.49
76.32
14.35
49.17
80.92
12:59
Imprewements
0.78
-0.70
2.72
2.11
0.77
4.17
-0.29
4.15
-4.95
0.25
-3.26
1.44
1.06
-6.33
9.69
MIXO
45.28
30.001
9023
30.31
79.37
0:33
19.33
07.25
10.00
$1.33
36.16
11.32
31.66
$9.30
B30
with our rog
$3.59
79.50
22:21
34.02
77.87
17.47
43.87
70.95
15.52
59.94
$3.95
52.82
56.54
85.21
23.52
Improvemettrs
5.31
1.11
13.18
3:45
1.70
11:34
4.32
3.75
4.91
8.59
2.21
21.00
4.88
4.29
15.45
GMLXI
4535
$1.41
0.2
31.60
K1.06
3038
40.30
60.30
11.19
51.15
87.35
3.32
$1.75
$6.87
11.30
with our rog
$1.96
79.74
18.15
52.53
79.15
12:52
43.90
69.02
16.99
56.31
83.31
25.45
55.11
87.67
17.59
Improvenettss
3.11
-1.65
8.92
1.24
-281
7.43
2.69
0.24
3.80
5.15
-.42
16.16
3.36
0.80
6.30
Table 22: MDG+GCD results: accuracy scores of each domain in VLCS dataset.
VLCCS Matheel
Asg
Caltech101
LabelMe Known
SUN19
VOC207
Known
Unkkown
All
Known
Tnknnwen
Unknowe
All
Knowen
Unknewn
Knows
Unknown
ERM
51.31
$2.89
33.35
$2.07
T3.37
33,83
30.34
92.01
4.35
62.07
95.15
11:38
31.35
69:31
37.35
with our reg
61.89
$1.53
36,85
16.59
75.13
77.36
50.54
91.02
6.13
60.70
92.83
11.48
58.02
66.35
51.63
Imgrevements
-2002
-1.55
1.77
-5.48
0.26
-5.49
0.09
-0.99
1.29
-1.37
-2.33
0.30
6.66
-3.16
14.18
PIM
63.19
50.34
40.24
50.39
32.05
$4.77
53.84
91.74
12.07
62.22
96.21
13:21
56.31
63.36
50.92
With our FOg
59.29
77.96
36.81
72.61
73.33
72.24
53.98
90.75
13:45
$6.85
85.64
15.81
53.72
64.13
45.75
Imgrovemenns
-3.90
-2.38
-3.45
-7.77
1.25
-12.53
0.14
0.99
1.38
-3.37
-10.57
2.50
2250
0.37
-3.16
MIRO
61.53
52.72
35.00
62.77
74,10
$7.33
51.81
91.53
7.72
62.22
95.50
11:09
49.32
69.34
33.99
With OUE rog
62.07
53.18
35.21
$2.51
74,62
$6.66
49.51
94.43
0,30
60.97
98.65
9.35
55.31
69.00
44.84
Improsements
0.56
0.46
0.18
-022
0.51
-0.67
-2.31
2.50
-7.72
-1.26
-0.94
-1.74
6.00
-0.34
10:85
GMDG
61.36
53.31
33.75
$2.51
74,87
56.52
49.93
95.24
0.30
59.86
93.96
7.62
53.13
69.17
40.8
with our mgg
62.33
52.77
36.09
$4.54
74.62
$0.76
49.98
92.01
3.56
61.33 30
95.03
9.84
53.
69 40.43
41.11
Improsements
0.97
-0.54
2.34
2,03
-0.35
3.23
0.05
-3.23
3.56
1.52
1.00
2.22
0.26
0.26
0.26
Table 23: MDG+GCD results: accuracy scores of each domain in TerraIncognita dataset.
Teralncagnis
Avg
Tacal
Local
Local 43
Toral
Method
All
Known
Unknown
All
Known
Unkzown
All
Known
Unknowa
All
Known
Unkzzww
All
Known
Unknown
ERM
37:34
20:46
23.15
46.51
1:25
57.07
30:85
25.25
44.11
29.41
24.55
4015
53.59
27.70
38.36
with OUE r0g
40.73
29:27
35.56
52.94
0.34
65.27
39.26
33.76
41.64
40.24
$7.45
1.03
30.47
25.71
34.32
Improvements
3.38
8:81
-9:58
6.43
-1.11
8.20
.0.62
5.55
-3.27
10.53
32:40
-39.22
-3.12
-1.99
-4.44
PIM
47.75
25.31
50.85
50.20
28.97
55.15
56.22
19.71
71.99
46.69
47.94
43.86
57.85
44.64
32.40
with OLE 20g
49.74
31.08
50.01
53.94
34.12
55.57
59.87
19.58
77.26
47.07
62:75
11:35
38,09
19.38
52.57
Improvemennt
2,00
1.23
0.84
3.74
5.15
3.41
3.05
0.13
5.28
0.38
14.82
32.51
0.21
24.76
20.47
MIRO
30.22
39.92
49.45
52.23
51.25
$2.46
55.54
14.73
73.16
48.93
62.89
17.13
44.19
30.79
55.06
With OLE 20g
48.85
20.87
38.42
36.26
22.42
64.15
31.27
23.75
34.52
$4,03
65.11
25.79
37.84
52.18
26.20
Improvementt
3.37
0.95
11.05
4,00
28.83
11.71
24.26
9.05
35.64
5.10
2:22
11.65
6.35
21.39
28.56
GMDG
32.38
40.07
16,70
35.65
20.86
49.46
61.76
21.47
58.33
with DUE 20g
45.36
39.77
41,55
51.89
38.16
55.00
09
41.30
22.05
49:00
30.47
65.29
16.72
39.77
33.59
46.99
Improwementts
5.83
7.39
1.49
15.19
2.51
18.15
4.61
1.19
6.05
1.01
3.55
-.75
2.53
22.34
13.54
25

---

Table 24: MDG+GCD results: accuracy of each domain in DomainNet dataset.
DomainNet
Avg
elipart
Info
painting
Known
Unknown
Unknown
ERM
22.56
40.89
6.85
31.04
58.32
7.15
17.94
34,71
6.85
30.59
51.82
9.34
reg
47.07
32.03
Improvements
3.31
6.18
0.34
0.99
0.11
1.76
0.23
-0.41
0.65
1.33
0.76
1.90
24.03
42.59
7.86
32.01
9.80
33.56
9.03
22.22
7,80
with our reg
24.66
43.86
7.78
31.91
57.76
9.26
16.99
30.77
7,89
28.17
45.94
10.37
Improvements
MIRO
31,49
55.44
10.57
40.13
67,55
16.11
25.84
48.53
10.84
37.89
62,45
13.29
reg
36.74
Improvements
0.10
00.77
0.41
0.49
-0.83
1.64
.0.26
-2.70
1,35
-1.15
-0.16
-2.14
GMDG
31.15
40.38
13.84
46.50
10:72
36.29
59
12.75
with our reg
31.75
55.18
11:30
40.91
68.17
17:05
26.60
49.11
11.71
36.82
60.76
12.85
Improvements
0.01
0.53
3.21
2.61
0.99
0.53
DomainNer Method
Avg
quickdraw Known
real
sketch
Known
Unknown
All
Unknown
Known
Unkkown
All
Known
Unknown
ERM
8.88
12.83
4.91
17.88
31.20
4,10
29.01
56.45
8.76
with reg
9.04
14.73
3.31
34.34
63.94
3.69
29.68
58.41
8.49
Impruvements
0.16
1.91
-1.59
16.45
32.74
.0.4
0.67
1.96
.0.27
9.92
14.73
5.09
29.09
53.88
3.42
32.12
59.35
12.03
with our reg
9.94
15.11
4.74
30.26
56.
3.47
30.68
57.43
10.95
Improvementts
0.02
0.38
-0.35
1.17
2.25
0.05
-1.44
-1.93
-1.08
MIRO with reg Impruvements
8.06
12.12
3.98
42.
75.49
7.72
34.83
66.51
11.46
9.36
15:73
2.95
42.00
74.36
8.50
35.23
64.89
13.34
1.30
3.61
-1.03
-0.20
-1.14
0.78
0.40
-1.62
1.88
GMDG with our reg Improvements
7.43
11.83
3.01
42.84
75.27
9.27
35.01
66.95
11.46
9.11
13.51
4.70
42.63
74,42
9.72
34,44
65.13
11.80
1.68
1.67
1.69
-0.21
-0.84
0.45
-0.58
-1.81
0.34
26

---

Unseen domain: Art painting
Without L-Reg
With L-Reg
Seen domain: Photo
Without L-Reg
With L-Reg
Seen domain: Sketch
Without L-Reg
With L-Reg
Seen domain: Cartoon
Without L-Reg
With L-Reg
Figure 7: GradCAM visualizations: Baseline is GMDG. The used dataset is PACS. The model trained under uDG+GCD setting with and without L-Reg, respectively. can be seen that for the known class 'dog.' the model trained with L-Reg extracts the area around the nose area for classification across seen and unseen domains.
27

---

Unseen domain: Art painting
Without L-Reg
With L-Reg
Seen domain: Photo
Without L-Reg
With L-Reg
Seen domain: Sketch
Without L-Reg
With L-Reg
Seen domain: Cartoon
With Without
L-Reg
L-Reg
Figure 8: GradCAM visualizations: Baseline is GMDG. The used dataset is PACS. The model is trained under uDG+GCD setting with and without L-Reg, respectively. It can be seen that for the known class 'elephant,' the model trained with L-Reg extracts the shape of long noses, teeth, and big ears for classification across all seen and unseen domains. The compromise the known sets can be seen in the sketch domain, where those features not significant.
28

---

Unseen domain: Art painting
Without L-Reg
With L-Reg
Seen domain: Photo
Without L-Reg
With L-Reg
Seen domain: Sketch
Without L-Reg
With L-Reg
Seen domain: Cartoon
Without L-Reg
shutterst
wrcck
With L-Reg
shutterst
Figure 9: GradCAM visualizations: Baseline is GMDG. The used dataset is PACS. The model trained under uDG+GCD setting with and without L-Reg, respectively. can be seen that for the known class 'giraffe,' the model trained with L-Reg extracts the feature of the long necks for classifying across all seen and unseen domains.
29

---

Unseen domain: Art painting
Without L-Reg
With L-Reg
Store
Seen domain: Photo
Without L-Reg
With L-Reg
Seen domain: Sketch
Without L-Reg
With L-Reg
Seen domain: Cartoon
Without
L-Reg
With L-Reg
Figure 10: GradCAM visualizations: Baseline is GMDG, The used dataset is PACS. The model is trained under uDG+GCD setting with and without L-Reg, respectively. It can be seen that for the known class 'guitar,' the model trained with L-Reg extracts the features of the necks and the strings of the guitar for classification across all seen and unseen domains.
30

---

Unseen domain: Art painting
Without L-Reg
With L-Reg
Seen domain: Photo
Without L-Reg
With L-Reg
Seen domain: Sketch
Without L-Reg
With L-Reg
Seen domain: Cartoon
Without
L-Reg
With L-Reg
Figure 11: GradCAM visualizations: Baseline is GMDG. The used dataset is PACS. The model is trained under uDG+GCD setting with and without L-Reg, respectively. It can be seen that for the unknown class 'horse,' the model trained with L.Reg extracts the features of the overall outline shapes of horses for classification across all seen and unseen domains.
31

---

Unseen domain: Art painting
Without L-Reg
With L-Reg
Seen domain: Photo
Without L-Reg
With L-Reg
Seen domain: Sketch
Without L-Reg
With L-Reg
Seen domain: Cartoon
With Without
L-Reg L-Reg
Figure 12: GradCAM visualizations: Baseline is GMDG. The used dataset is PACS. The model trained under uDG+GCD setting with and without L-Reg, respectively. can be seen that for the unknown class 'person," the model trained with L-Reg extracts the features of human faces for classification across all seen and unseen domains. The compromise of the known sets can be seen in the sketch domain, where those faces are not drawn.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper discusses the limitations of the work in the Conclusion Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

Yes, the paper provides the full set of assumptions and complete proof in the main paper and appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper provides all the information needed reproduce the main ex- perimental results of the paper much possible in the appendix and supplementary
materials.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
34

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The paper provides uses open- open-source data and open access to the code. The code is especially attached as supplementary material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, this paper specifies all the training and test details the Experiments Section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Following previous work, this paper does not report error bars. However, this paper reports some results in the form of mean with standard deviation, shown in Table However,
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
35

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides information on the computer resources in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eeuripp..ee cttss.ccppubbll ctc/puublc/thiicccuueee.ll..
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We think that there is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
36

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper seems to pose no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets (c.g., code, data, models) used in the paper properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
37

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papepswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: New assets introduced in the paper are documented, and the documentation is provided alongside the assets. The model training code provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
38

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "SRPmUpeicD", "reviewer": "Reviewer_y1GK", "summary": "This paper mainly focuses on two problems: 1) How does logical reasoning relate to visual tasks such as image classification? 2) How can we derive a logical reasoning-based regularization term to benefit generalization?. Then, this paper proposes a method called Logical Reasoning Regularization based on the analysis of the two problems. Theoretical analysis and experimental results demonstrate that L-Reg enhances generalization across several scenarios.", "questions": "1. line 123, f_s should be f.\n2. In Figure 1, it can be seen from the visualization results of the first line that the learned features are basically concentrated on the background, and it is felt that the obtained model is overfitting or underfitting. I would like to know the specific parameter Settings, experimental codes, random seeds, etc., of the experiment.\n3. In Figure 3, what does the abscissa represent? What is the baseline being compared? To make the results more convincing, an additional baseline and dataset need to be added.\n4. In Figure 4, the coordinate proportions of the left subgraph and the right subgraph are inconsistent, so it is unfair to make a direct comparison. Secondly, the horizontal and vertical coordinates need to be explained what they represent. Finally, the left and right subgraphs are almost identical, It cannot be proved that +L-Reg can achieve real elimination of certain extracted semantics characterized by dominant frequencies across all samples.\n5. In lines 173-186, Why the constraint in eq. (6) is removed in eq. (8), please give a detailed derivation process.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The main contributions of this article are: 1) Building the relationship between logical reasoning and visual tasks such as image classification; 2) Rethinking the classification task from the logical reasoning perspective and proposing Logical Reasoning Regularization. Overall, the contributions are meaningful, and the paper is interesting.\n2. The paper is easy to read.", "weaknesses": "1. As can be seen from Table 1 and Table 2, the proposed regular term has a weak improvement on the existing method.\n2. The analysis of this paper is incomplete and lacks theoretical analysis of sufficient conditions or sufficient and necessary conditions for meeting Atomic Formulas.\n3. Overclaim. This paper claims that the proposed L-Reg can reduce the complexity reduction. However, the proposed L-Reg is regarded as a regular term and is directly added in the learning objective. This does not reduce the computational complexity of the training period, but the extra regular term increases the computational overhead of the whole training process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and we address your weaknesses and questions point-by-point. \n\n**W1.** We humbly believe L-Reg delivers consistent and evident gains. Please refer to Reply to All for the highlighted improvements. **Moreover, as you suggested, we further validate L-Reg's efficacy by applying it to the ERM for the mDG and CircuitFromer for additional Congestion prediction. PDF Tab.4 exhibits that L-Reg improves averaged performance with the ERM baseline from 49.9% to 52.9% and CircuitFormer from 0.6374 to 0.6553 in the Pearson metric, besides increases in other metrics.**\n\n**W2.** The sufficient and necessary conditions for achieving atomic formulas are the ultimate goal that remains a problem for the community. While we are working towards this, this paper proposes a first practical approach to approximate the most general atomic formulas. Please refer to the Reply to All, which provides more analysis about how the constraints in the paper are derived to achieve atomic formulas. Furthermore, according to the current limitation of L-Reg the paper discussed, we offer a future direction of obtaining proper $z$ to meet L-Reg's precondition. Additional experiments of ERM on mDG and GCD (cf. PDF Tab. 2,3&4) show that constraining proper $z$  leads to further improvement. We will discuss this grand topic in our revision and explore more aspects in the future. \n\n**W3.** Sorry for any confusion. L-Reg is not designed to reduce the computational complexity but aims to reduce the complexity of the model parameters and the data features. As discussed in Paper Sec.3, L-Reg reduces the classifier's complexity by increasing the sparsity of its weights and features' complexity by removing over-dominated semantics across all classes (this comment is related to your Q3\\&4, where details of how L-Reg achieves these can be found). We will polish this part in the final version for better clarity. \n\n**Q1.** Thanks. We will fix all these typos.\n\n**Q2.** Paper Fig.1 shows CAM visualizations of the models trained under the GMDG with the RegNetY-16G backbone for mDG+GCD on the PACS with the unseen domain art painting, using DomainBed's protocols and codes. Both models share the same training parameters and seed 0. The only difference between them is that the latter uses L-Reg with its weight as an extra hyperparameter. Details such as specific parameter settings, as you requested, are in Paper Appendix E.1 and codes for our method are included in the supplementary materials. In short, we believe the comparison should be fair because of the same hyperparameters and the seed being used for both models except  L-Reg. Note that the same models are used for Paper Fig.3&4, and all CAM visualizations in the Paper appendix. \n\n**Q3.** Using the aforementioned models, Paper Fig.3(a) presents the heatmap of the classifiers' weights, where the x-axis is the index of the weights in the linear layers. Paper Fig.3(b) shows the distributions of the classifiers' weight values, with the x-axis representing the value of the normalized weights and the y-axis showing the count of weight values in bin intervals. Following your suggestion, the fixed figure with a denoted abscissa and more descriptive captions is shown in PDF Fig.2. PDF Fig.2 demonstrates that L-Reg reduces the classifier's complexity by alleviating extreme weight values. The heatmap further indicates that L-Reg increases the sparsity of the classifiers, thus better generalization. Please refer to the reply of **W1** for details of more experimental results.  \n\n**Q4.** We apologize for any confusion caused by Paper Fig.4. We have re-drawn Paper Fig.4, now shown as PDF Fig.1, with distributions illustrated on the same coordinates and added features of known and unknown classes. Using the aforementioned models, PDF Fig.1 shows feature distributions based on the first component values after PCA (denoted as $v1^{st}$). Before using L-Reg, $v1^{st}$ mostly concentrates between [-0.4,-0.2] across all classes, indicating some specific semantics over-dominates the features. L-Reg alleviates this issue by forcing the model to obtain each class's minimal semantic supports, removing shared semantics across all classes to reduce feature complexity. PDF Fig.1  top row indicates the distance between the feature distributions of known and unknown classes is enlarged with L-Reg, making them more dividable for classification.\n\n**Q5.** We provide more details about why $\\vdash_{(h \\circ g(X), Y)}=\\models_{(g(X_s),Y_s)}$ in Paper Eq.6 can be safely omitted in the rest of the paper after the definition of logical framework. Consider the logic $L_{(X_s,Y_s)}=<F_{(X_s,Y_s)},D,\\models_{(X_s,Y_s)},h,\\vdash_{(h(X),Y)}>$, we want to study $\\vdash$'s logic that is defined in the form of\n$L_{\\vdash}\\stackrel{\\text{def}}{=}<F_{X_s,Y_s},D_{\\vdash},h_{\\vdash},\\models_{\\vdash}>$, where $D_{\\vdash},h_{\\vdash},\\models_{\\vdash}$ are pseudo-components associated with $\\vdash$. Particularly, $D_{\\vdash}$ is a subset of all possible world/domains from $F_{(X_s,Y_s)}:D_{\\vdash}\\stackrel{\\text{def}}{=}\\{T\\subseteq F_{(X_s,Y_s)}:T\\text{ is closed under }\\vdash_{(h(X),Y)}\\}$. For any $T\\in D_{\\vdash}$ and $a\\in F_{(X_s,Y_s)}$, it has $h_{\\vdash}(a,T)\\stackrel{\\text{def}}{=}\\{b\\in F:T\\vdash(a\\leftrightarrow b)\\}$. Further, $\\models_{\\vdash}$ in $T\\in D_{\\vdash}$ is defined as $T\\models_{\\vdash}a\\stackrel{\\text{def}}{\\Leftrightarrow}a\\in T$. [1] points out that the following condition is almost always satisfied: (Cond) $\\forall a, b\\in F_{\\vdash},d\\in D_{\\vdash}$, we have $(h_{\\vdash}(a,d)=h_{\\vdash}(b, d))\\text{ and }d\\models_{\\vdash}a\\Rightarrow d\\models_{\\vdash}b$. Hence, the semantical consequence relation induced by $\\models_{\\vdash}$ coincides with the original syntactical $\\vdash_{(h\\circ g(X),Y)}$ while Cond holds. Due to $D_{\\vdash}\\subseteq D$, $\\models_{(g(X_s),Y_s)}$ coincides with $\\models_{\\vdash}$. Thus, $\\vdash_{(h\\circ g(X),Y)}=\\models_{(g(X_s),Y_s)}$ can be safely omitted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CCKsG0rgWm", "reviewer": "Reviewer_8YRB", "summary": "The paper proposes a novel logical regularization termed L-Reg for visual classification. L-Reg encourages models to focus on the salient semantics and thereby emerges interpretability. The theoretical analysis provides clear connections between logical reasoning and L-Reg. Extensive experiments demonstrate that L-Reg also benefits the generalization of models to unseen domains and categories.", "questions": "See weakness.", "limitations": "The author have discussed both the limitations and potential societal impact. The limitations are left for future research.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tStudies on loss regularization have positive influences on various fields.\n2.\tThe paper is well-presented, and L-Reg is clearly presented with rigorous theoretical analysis.\n3.\tThe qualitative benefits of L-Reg are validated through experiments, and the generalization brought by L-Reg has been demonstrated sufficiently under three settings.", "weaknesses": "1.\tAs stated in the introduction section, interpretability is a longstanding focus among studies in regularization terms. The authors claim that L2 regularization might lead to ambiguous interpretability, which also serves as an important motivation and contribution for L-Reg. However, the interpretability of L-Reg has not been adequately discussed except for the introduction section. Further analysis from either qualitative or quantitative perspectives can greatly strengthen the presentation.\n2.\tThe analysis of Figure 5 should be detailed, especially the examples of unknown classes in Row 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "These insightful comments are highly appreciated. We believe these two questions are very related; therefore, please allow us to address them together. \n\nAs discussed in  Reply to All Reviews 1, the L-Reg's interpretability is rooted in learning the good general atomic formulas. Specifically, L-Reg encourages the model to identify the minimal semantic supports - the most important features -  necessary for class recognition. Such an approach resembles humans' cognition process. Paper Fig.1,5 and visualizations included in the Paper appendix show that L-Reg enables the model to learn distinctive features such as facial features for recognizing the person class, the long-neck feature for the giraffe class, and so on. \n\nTo address your questions more concretely, we use Paper Fig.5 as a typical example to analyze L-Reg’s interpretability further. \n\nFor the known classes, the efficacy of L-Reg can be intuitively understood as extracting the minimal semantic supports for a given class label. As examples shown here, the presence of a guitar's fingerboard, even in unseen domains, helps classify a sample as belonging to the guitar category, whose informal form can be denoted as $h(F_{(\\gamma=\\text{has the fingerboard},y=\\text{guitar})},d\\in D)\\rightarrow\\text{True}$.  For all known classes, samples with these minimal semantic supports are recognized accordingly.\n\nIn contrast, if a sample lacks these minimal supports for any known class, it is very likely categorized as an unknown class. This behavior stems from Paper Eq.10 which ensures $\\mathcal{A}^{y_i*} \\neq \\mathcal{A}^{y_j*}$ through constraining $\\gamma^{y_i} \\neq \\gamma^{y_j}$. L-Reg further enhances the model's ability to identify minimal supports for unknown classes by filtering out co-covariant features associated with other classes and thus generalizing to unseen domains. Therefore, the very interpretable features for unknown classes from unseen domains can be extracted using L-Reg. Paper Fig.5 (right side) demonstrates that the model with L-Reg can even extract facial features for the unknown person class and can generalize this to the unseen domain. Similarly, here we obtain an (informal) atomic formula as $h(F_{(\\gamma=\\text{has a human face},y=\\text{person})},d\\in D)\\rightarrow\\text{True}$.\n\nHowever, as shown in Row 3, significant domain shifts, such as those between the sketch domain and other domains, pose challenges. Specifically, the differences between the stick-figure style of sketches of persons and figures from other domains can hinder the model's ability to cluster sketches with other domains' figures when the class label is unknown. Thus, under this circumstance, the model may fail to extract meaningful features from those sketches. We acknowledge this limitation and will explore solutions in future work.\n\nOnce again, we appreciate your thoughtful feedback. We will incorporate this analysis into the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EiZIdyzjvJ", "reviewer": "Reviewer_t6Vg", "summary": "This work introduces a sample-based regularization technique, L-Reg, which goes beyond techniques like parameter-based L2 regularization by being more interpretable and demonstrating better generalization ability. The work formalizes the notion of semantic support to force the model to learn minimal sufficient statistics, quantitatively and qualitatively showing how that it leads to better generalization across multiple settings - multi-domain generalization, generalized category discovery and a new setting that is a combination of the two which they introduce.", "questions": "* Since the paper mentions “constructing atomic formulae”, would it be possible to actually extract and see how these look from the model? This might significantly strengthen the paper.\n* How does this method compare, formally and empirically, with sparse concept-based models? Wouldn’t their loss be very similar to the proposed L_reg loss?", "limitations": "Yes, limitations have been addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ A theoretically grounded paper with comprehensive experiments and results. \n+ The paper is well written in general. \n+ I especially liked how the algebraic logic formalism was neatly tied into this space. The idea of using semantic supports, although simple, is motivated and formalized well. I also liked how the negation of the semantic support set was used to formulate the optimization problem. \n+ The derivation of conditions required to hold under various settings is well done and makes the derivation of the regularization easy to follow. The proposed mDG + GCD setting is interesting.", "weaknesses": "* Although the method introduced is technically sound, with the baselines being quite comprehensive, the improvement in results seems minor. This suggests that accuracy may not be the right metric to compare against here. Considering the objectives proposed, shouldn’t other metrics beyond accuracy be considered?\n* The paper states that “the semantics generated by the encoder and classifier can be combined to form atomic formulas”. I expected to see some of the actual learnt atomic formulae in the results - which I did not.\n* From the formulation of the L_reg loss, it appears that any concept-based model that is sparse may be similar to the proposed formulation. How is the proposed method different from such methods -- formally and empirically?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments, and we address your weaknesses and questions point-by-point. Some points of weaknesses and questions are combined because they are very associated with each other. \n\n**W1.** We really appreciate this insightful comment. In our study, we adopted the commonly used accuracy metric to align with the previous work we compared against. We agree that other entropy-based metrics could potentially provide a more effective evaluation. Additionally, it may be worthwhile to propose novel logic-based metrics for a more comprehensive assessment. We find this to be a very attractive and promising area of research, and we are eager to explore it further in future work.\n\n**W2&Q1.** We appreciate these comments. For a more detailed explanation of L-Reg's interpretability, please refer to Reply to All Reviews 1. Notably, the CAM visualizations in our paper illustrate the model's learned atomic formulas. By using L-Reg, the model derives interpretable atomic formulas, which can also be understood as the most important features for predicting a given class. To address your questions more concretely,  we use Paper Fig.5 as a typical example to analyze L-Reg’s interpretability further. \n\nFor the known classes, the efficacy of L-Reg can be intuitively understood as extracting the minimal semantic supports for a given class label. As examples shown here, the presence of a guitar's fingerboard, even in unseen domains, helps classify a sample as belonging to the guitar category, whose informal form can be denoted as $h(F_{(\\gamma=\\text{has the fingerboard}, y=\\text{guitar})},d\\in D)\\rightarrow\\text{True}$. For all known classes, samples with these minimal semantic supports are recognized accordingly.\n\nIn contrast, if a sample lacks these minimal supports for any known class, it is very likely categorized as an unknown class. This behavior stems from Paper Eq.10 which ensures $\\mathcal{A}^{y_i*} \\neq \\mathcal{A}^{y_j*}$ through constraining $\\gamma^{y_i} \\neq \\gamma^{y_j}$. \nL-Reg further enhances the model's ability to identify minimal supports for unknown classes by filtering out co-covariant features associated with other classes and thus generalizing to unseen domains.\nTherefore, the very interpretable features for unknown classes from unseen domains can be extracted using L-Reg. Paper Fig. 5 (right side) demonstrates that the model with L-Reg can even extract facial features for the unknown person class and can generalize this to the unseen domain. Similarly, here we obtain an (informal) atomic formula as $h(F_{(\\gamma=\\text{has a face}, y=\\text{person})},d\\in D)\\rightarrow\\text{True}$.\n\nWe will include a more detailed discussion on this topic in the final version of our paper.\n\n**W3&Q2.** We really appreciate this inspiring comment. Following the Reply to All part 1, while a common sparse concept model may be able to achieve $\\gamma^y\\psi = z^y$ by filtering irrelevant features through the sparsity, it may not ensure $\\gamma^{y_i} \\neq \\gamma^{y_j}$, which is crucial for disentangling features used for predicting different classes. This limitation can potentially lead to degradation in generalization performance for common sparse concept models.\n\nTo investigate this fairly, we re-implemented the Bernoulli Sample of the latent features from the Sparse Linear Concept Discovery Models [3] on the same PIM backbone that we used to achieve the sparsity.  The results in PDF Tab.2&3 indicate that while L-Reg consistently achieves overall improvement,\nthe sparse concept-based approach does not consistently improve generalization, validating the aforementioned difference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sKvcjbCGwN", "reviewer": "Reviewer_BLvb", "summary": "This paper addresses the multi-domain generalization (mDG), generalized category discovery (GCD), and the more challenging mDG+GCD task. The authors introduce a logical reasoning-based regularization term called L-Reg, which bridges logical analysis with image classification to enhance model interpretability and reduce complexity. The main idea of L-Reg is to identify a minimal set of semantics that can deduce the relationship between $x$ and $y$, i.e., the semantic support. Theoretical analysis and experiments demonstrate that L-Reg improves generalization across mDG, GCD, and mDG+GCD.", "questions": "1. Line 145 claims that \"Semantics that occur frequently across samples often lack decisiveness for classification.\" However, samples from the same class often share the semantic predictive of the class. \n2. The definition of the \"minimal semantics\" $z_i$ is vague. The meaning of $Z_i$ is also unclear in Eq. (3).\n3. Could you provide a concrete implementation of L-Reg? I wonder which layer(s) is(are) chosen for computing Eq. (3) in practice? Are the results sensitive to the selection of layers?", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper establishes a connection between logical reasoning and practical visual generalization problems, bringing novel insights for improving DG and GCD.\n2. The proposed L-Reg used over existing SOTA methods can further improve the SOTA performance.\n3. The experiments are comprehensive. The visualization results provide a good empirical understanding of the role of L-Reg.", "weaknesses": "1. The presentation of this article is somewhat confusing for readers unfamiliar with logical reasoning, especially from Section 3.1 to Section 3.2. It would be better to add intuitive explanations of key concepts. For example, why should $F$  be atomic formulas for a logic to be a 'good general' one, or, what do the atomic formulas refer to in a real-world case?\n2. The mDG improvements of L-Reg are over GMDG. The performance of directly applying L-Reg is not shown, i.e., ERM+L-Reg.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your insightful comments, and we address your weaknesses and questions point-by-point. Some points of weaknesses and questions are combined because they are very associated with each other.\n\n**W1&Q2.** Thank you so much for your comments, and sorry for any confusion caused by our writing.  We will follow your kind suggestions and add some intuitive explanations for key concepts in our paper. Please also kindly refer to Reply to All Reviewers for the theoretical analysis of atomic formulas and their relationship to the interpretability of L-Reg.\nTo address your questions more concretely, we use Paper Fig. 5 as a typical example to provide more analysis on atomic formulas and the interpretability of L-Reg.\n\nFor the known classes, the efficacy of L-Reg can be intuitively understood as extracting the minimal semantic supports for a given class label. As examples shown here, the presence of a guitar's fingerboard, even in unseen domains, helps classify a sample as belonging to the guitar category, whose informal form can be denoted as $h(F_{(\\gamma=\\text{has the fingerboard}, y=\\text{guitar})},d\\in D)\\rightarrow\\text{True}$. For all known classes, samples with these minimal semantic supports are recognized accordingly.\n\nIn contrast, if a sample lacks these minimal supports for any known class, it is very likely categorized as an unknown class. This behavior stems from Paper Eq.10 which ensures $\\mathcal{A}^{y_i*} \\neq \\mathcal{A}^{y_j*}$ through constraining $\\gamma^{y_i} \\neq \\gamma^{y_j}$. L-Reg further enhances the model's ability to identify minimal supports for unknown classes by filtering out co-covariant features associated with other classes and thus generalizing to unseen domains. Therefore, the very interpretable features for unknown classes from unseen domains can be extracted using L-Reg. Paper Fig. 5 (right side) demonstrates that the model with L-Reg can even extract facial features for the unknown person class and can generalize this to the unseen domain. Similarly, here we obtain an (informal) atomic formula as $h(F_{(\\gamma=\\text{has a face}, y=\\text{person})},d\\in D)\\rightarrow\\text{True}$.\n\nWe will include a more detailed discussion on this topic in the final version of our paper.\n\n**W2.** Thank you very much for this comment. Please refer to the Reply to All Reviewers for the experimental details on the ERM baseline.  As shown in PDF Tab.4, under the same experimental settings and hyperparameters, incorporating L-Reg with ERM significantly enhances the overall mDG performance, improving it from 49.9% to 52.9%.\n\n**Q1.** We apologize for any confusion caused and appreciate your constructive feedback. L-Reg is designed to focus on eliminating shared semantics across all classes rather than addressing frequent features within a single class. To clarify this,  please refer to the PDF Fig.1. In this figure, we have included fixed images with the same coordinates and additional figures illustrating feature distributions of known and unknown classes. Note here we use the values of the first components of PCA results on the original features, denoted as $v1^{st}$.\n\nBefore applying L-Reg, the $v1^{st}$ feature predominantly falls within the range [-0.4, -0.2] across all classes, especially identical between known and unknown classes. This indicates that a few specific semantics overly influence many features. L-Reg mitigates this issue by focusing the model on disentangled minimal semantic supports for classifying each class, thereby reducing feature complexity and enhancing generalization.\n\n**Q2.** Many thanks for this comment. \nThe code for L-Reg is available in the supplementary materials we have provided, and we will release all code and hyperparameters to facilitate the reproduction of our experiments.\n\nRegarding sensitivity, as discussed in Paper Line 344-358  and illustrated in Paper Tab.4, applying L-Reg to the semantic features from the deep layers improves performance for unknown classes without negatively impacting known classes. \nWe hypothesize this is due to the fact that our L-Reg is derived based on the precondition that $z^i, z^j \\in z, I \\neq j$ is independent of each other. This condition holds for most deep-layer features but may not apply to shallow layers, and further regularizing the independence may lead to further improvements.\nTo test this hypothesis, we conducted experiments with orthogonality regularization (Ortho-Reg) to enforce feature independence in mDG and GCD tasks.  As shown in PDF Tab.2,3&4, while using Ortho-Reg alone may not be very effective, combining L-Reg with Ortho-Reg leads to further improvements.\nThese findings support our hypothesis and suggest that L-Reg, particularly when applied to deep layers or in conjunction with Ortho-Reg, is beneficial.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bCdmaws8k3", "reviewer": "Reviewer_2Pp3", "summary": "The paper introduces a logical regularization method called L-Reg, aimed at enhancing the generalization ability of image classification tasks through a logical reasoning framework. L-Reg effectively simplifies the complexity of the model by ensuring that the generated atomic formulas align with the logical relationships between the images and labels, promoting balanced distribution in the feature space, and reducing the number of extreme weight values in the classifier. Both theoretical analysis and experimental results validate the effectiveness of L-Reg in various generalization scenarios, especially demonstrating outstanding performance in multi-domain generalization and generalized category discovery tasks.", "questions": "I hope the authors can further compare L-Reg with other regularization methods in terms of generalization ability and interpretability to highlight the advantages of L-Reg.", "limitations": "Authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. L-Reg effectively reduces the complexity of the model by balancing the feature distribution and reducing the number of extreme weight values in the classifier.\n\n2. In Sections 3.1 and 3.2, the paper provides logic-based theoretical analysis and detailed derivation of the construction process of L-Reg. Furthermore, through experiments in Sections 4 and 5, the effectiveness of L-Reg in different generalization settings, especially in multi-domain generalization and generalized category discovery tasks, is validated.\n\n3. Designed as a plug-in loss function, L-Reg is compatible with most existing frameworks, making it highly flexible and practical in real-world applications.\n\n4. The paper also explores the relationship between logical reasoning and visual classification tasks, delving into the derivation of logic-based regularization terms to promote generalization, providing new perspectives and methods for research in related fields.", "weaknesses": "1. L-Reg may reduce the scope of semantic support, leading to a slight performance decrease on known datasets. It is hoped that the authors can analyze in more detail the reasons for this performance drop and provide possible improvement methods.\n\n2. The appropriate function g for generating atomic formulas is a key factor. While the authors propose L-Reg as a regularization method to ensure that F(g(Xs), Ys) consists of atomic formulas, they do not elaborate on how to choose or design this function g. Further explanation on the specific methods or criteria for selecting g is desired.\n\n3. In multi-domain generalization tasks, how L-Reg integrates with existing methods and how to adjust alpha to balance the two losses are details that the authors are encouraged to further elucidate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your insightful comments, and we address your weaknesses and questions point-by-point. \n\n**W1.** Thank you for your insightful comment. As discussed in Paper Lines 344-358, L-Reg relies on the precondition that each dimension of the semantic features represents independent semantics. When this condition is not met, applying L-Reg can lead to performance degradation. This is because the atomic formulas constructed for different classes could become sub-optimal if the minimal semantic supports for different classes correlate with each other. In cases where improper $z$ is used, especially between known and unknown classes, the model may struggle to effectively filter out irrelevant features for unknown classes, which can result in features that inadvertently overlap with the minimal semantic supports for known classes, resulting in degradation for them. \n\nTo address this issue, we hypothesize that enforcing independence between $z^i$ and $z^j$ could lead to further improvements. To test this hypothesis, we conducted experiments using orthogonality regularization (Ortho-Reg) to enforce feature independence in mDG and GCD tasks. As shown in PDF Tab.2,3&4, results indicate that while Ortho-Reg alone may not be very effective, combining L-Reg with Ortho-Reg leads to significant improvements.\n\nBased on these findings, we propose that the performance drop observed could be mitigated further with a well-designed model architecture or additional regularization techniques that enhance independence between $z^i$ and $z^j$. We find this to be a very attractive and promising area of research, and we are eager to explore it further in future work.\n\n**W2.** We appreciate your comment. To ensure a fair comparison and align with previous work, we use the same encoder, $g$, as employed by earlier studies to validate our L-Reg approach. Insights into the selection or design of $g$ can be found in our response to W1.\nAs noted, L-Reg relies on the condition that the semantic features $z^i$ and $z^j$ (for $i \\neq j$) are independent. Models that achieve an orthogonal semantic space are thus well-suited for applying L-Reg, as they naturally align with this requirement.\n\n\n**W3.** hank you for highlighting this point; it has been very insightful. Currently, we use a plain strategy of selecting the regularization weights in the range of [0.01, 0.001, 0.0001], keeping them relatively small compared to the scale of other losses (approximately 1:10). We appreciate your comments and acknowledge that there may be deeper theoretical or empirical insights related to this issue. We plan to explore this further in future research.\n\n\n**Q1.** Thank you for your recommendation.\nFor a thorough evaluation, we compare our L-Reg to the aforementioned Ortho-Reg and a sparsity-based regularization approach. To validate this fairly, we re-implement the Ortho-Reg and Bernoulli sample of the latent features from the sparse linear concept discovery models [3] on the same PIM backbone that we used. The results, presented in PDF Tab.2\\&3, indicate that L-Reg consistently yields the most significant improvements when these regularization terms are applied alone. Additionally, as mentioned, combining L-Reg with Ortho-Reg further enhances performance since a more proper $z$ is obtained.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Reply to All**

We sincerely appreciate the reviewers' insightful comments, which have helped us refine and improve our paper. We have identified common concerns across the reviewers and address them collectively here. Detailed responses to individual reviewers are provided separately. Please note: References to contents from the paper are denoted with the prefix 'Paper'; 'PDF' indicates that the content is included in the uploaded PDF file.

**1. More about atomic formulas and interpretability of L-Reg**

Thank you for your comments regarding atomic formulas, which have guided us in highlighting the significance of L-Reg more effectively. The atomic formula $\mathcal{A}^y$ is of the form $h(f(g(x)),y,d)$ or $h(f(z, y),d)$. Our aim is to find the good (most) general $\mathcal{A}^{y*} \in \mathcal{A}^y$ for $y$ class from which the interpretability of L-Reg is derived.

Consider $\mathcal{A}^y_1, \mathcal{A}^y_2 \in \mathcal{A}^y$, if $\mathcal{A}^y_1$ is more general than $\mathcal{A}^y_2$, there will be a substitution $\psi$ such that $\mathcal{A}^y_1\psi=\mathcal{A}^y_2$ [1,4]. $\mathcal{A}^{y*}$  should meet $\mathcal{A}^{y*}\psi=\mathcal{A}^y_i\in \mathcal{A}^y$, which infers that $\gamma^y\psi = z^y$ (cf. Paper Eq.9) for predication of $y$ where $\gamma^y$ is the **semantic support** (cf. Paper Def.3.1). Note here that the form of $\mathcal{A}^y$ is constructed for $y\in Y$, i.e., predicate whether the sample belongs to the $y$ class. Considering multiple classes $y_i,y_j\in Y, i\neq j$, it has $\mathcal{A}^{y_i*}\neq \mathcal{A}^{y_j*}$ thus $\gamma^{y_i} \neq \gamma^{y_j}$ (cf. Paper Eq.10), which constrains that different minimal semantic supports should be used for predicting different classes. 

The interpretability of L-Reg is based on $\mathcal{A}^{y*}$, compelling the model to use distinct minimal semantic supports for each class.  These minimal semantic supports can be interpreted as the most critical features for efficient prediction. For example, as shown in Paper Fig.1, the model with L-Reg has learned the facial features of the person class (see more examples in Paper Supp Fig.7-12), forming the (informal) atomic formula $h(F_{(\gamma=\text{has a human face},y=\text{person})},d\in D)\rightarrow\text{True}$. 

**2. Improvements with L-Reg**

**Improvement highlights of L-Reg.**  We understand the points from the reviewers. Nonetheless, we humbly believe L-Reg actually leads to consistent and evident gains. Paper Tab 1-2 show the consistent overall improvements brought by L-Reg across different datasets in mDG and GCD, suggesting the feasibility of L-Reg. For GCD, **a 6.7% improvement on the unknown classes** of CIFAR100 and an average of 2.8\%  across unknown classes and all datasets also addresses L-Reg's efficacy for generalization. In mDG, the TerraInc dataset includes challenging camera trap images even for humans; L-Reg achieves **a significant 2.2% increase** on it and an average of 0.7\% across five datasets. 

**Apply L-Reg to ERM Baseline for mDG.** To further validate L-Reg's efficacy, we use ERM as the baseline on the TerraInc dataset for mDG. For a fair comparison, all experiments share the same hyperparameter settings and use the Regnety-16gf backbone. Original ERM results are also included alongside our reproduced results. The results in PDF Tab.4 reveal that  ERM  with L-Reg significantly improves mDG performance (**from 49.9% to 52.9%**).

**Apply L-Reg to congestion prediction for circuit design.** We also test L-Reg in Congestion prediction for circuit design on the CircuitNet [2] dataset by using the CircuitFormer [5] backbone. All parameters, except for L-Reg, remain consistent with CircuitFormer, and we follow its metrics. Results in PDF Tab.1 show improvements with L-Reg across all metrics and a significant increase in the pearson metric (**0.6374 to 0.6553**).

**Compare L-Reg with more regularization terms.** We compare L-Reg with other regularization terms: Ortho-Reg: the orthogonality regularization that constrains the independence of each dimension of the semantic feature $z$; and Sparsity: implemented as Bernoulli Sample of the latent features from the sparse linear concept discovery models [3] on our used PIM backbone. PDF Tab.2&3 demonstrate that L-Reg outperforms Ortho-Reg and Sparsity.

**Limitation of L-Reg and possible solutions.** As discussed in the Paper Limitation part and analysis around Paper Line 344-358, L-Reg is based on the precondition that each dimension of the $z\in Z$ represents an independent semantic. Thus, improper $z$ that does not meet this precondition may lead to sub-optimal results. To validate this hypothesis, we test L-Reg by reinforcing independence with Ortho-Reg. Results of MDG in PDF Tab.4 and GCD in Tab.2&3 show that combining L-Reg with Ortho-Reg leads to further improvements, whereas Ortho-Reg alone may not guarantee improvements. This suggests a direction for future work.

In summary, we believe the consistent improvements across all these experiments under different settings, with various baselines and backbones, demonstrate the excellent efficacy of L-Reg. These additional analyses and experiments will be included in the final version.


*References:*

[1] H. Andréka, I. Németi, and I. Sain. Universal algebraic logic. Studies in Logic, Springer, 2017.

[2] Z. Chai, Y. Zhao, W. Liu, Y. Lin, R. Wang, and R. Huang. Circuitnet: An open-source dataset for machine
learning in vlsi cad applications with improved domain-specific evaluation metric and learning strategies.
IEEE TCAD, 42(12):5034–5047, 2023.

[3] K. P. Panousis, D. Ienco, and D. Marcos. Sparse linear concept discovery models. In ICCV, pages 2767–2771, 2023.

[4] I. Tsapara and G. Turán. Learning atomic formulas with prescribed properties. In Proceedings of the
eleventh annual conference on Computational learning theory, pages 166–174, 1998.

[5] J. Zou, X. Wang, J. Guo, W. Liu, Q. Zhang, and C. Huang. Circuit as set of points. NeurIPS, 36, 2024.

Meta-review: **Summary:** This work introduces a sample-based regularization term, L-Reg. It goes beyond techniques like parameter-based L2 regularization by being more interpretable and demonstrating better generalization ability. It encourages models to focus on the salient semantics and thereby emerges interpretability. The theoretical analysis provides clear connections between logical reasoning and L-Reg.

**Strength:** The contributions --- building the relationship between logical reasoning (regularization) and visual tasks and the studies on loss regularization --- are meaningful; the paper is interesting and well-presented with rigorous theoretical analysis and comprehensive experiments and results; the idea of using semantic supports is motivated and formalized well; the derivation of required conditions is well done and makes the derivation of the regularization easy to follow; the proposed method is compatible with most existing frameworks, making it highly flexible and practical in real-world applications.

**Weakness:** Weak improvement empirically (other metrics might be presented); the interpretability of L-Reg has not been adequately discussed except for the introduction section; the analysis is incomplete and lacks theoretical analysis of sufficient (and necessary) conditions or sufficient and necessary for meeting Atomic Formulas; some arguments seem to be overly claimed; further analysis from either qualitative or quantitative perspectives and further discussions about sparse concept-based models can greatly strengthen the presentation; the writing could be improved for readers unfamiliar with logical reasoning; L-Reg may reduce the scope of semantic support, leading to a slight performance decrease on known datasets; some technical details and explanations are not clear or missing.

**After rebuttal:** The authors provided a rebuttal, including new experimental results. All the reviewers read the rebuttal and provided feedback. The reviewers have acknowledged that most of their concerns are well-addressed. **After rebuttal, the paper received increased ratings 6-6-6-6-6, with an average rating of 6.**

**Recommendation:** The AC checked the review, rebuttal, and the reviewers’ further feedback. The AC agreed with the reviewers about the multiple strengths of the papers and appreciated the authors’ efforts to effectively address weaknesses. The AC thus recommended **acceptance** and suggested the authors incorporate all the reviewers’ comments and their rebuttal into their final version (especially Reviver t6Vg’s and Reviewer 2Pp3’s comments).
Decision: Accept (spotlight)