Title: Interpret Your Decision: Logical Reasoning Regularization for Generalization in Visual Classification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Domain generalization

generalized category discovery

image classification

logical reasoning
Abstract: Vision models excel in image classification but struggle to generalize to unseen data, such as classifying images from unseen domains or discovering novel categories. In this paper, we explore the relationship between logical reasoning and deep learning generalization in visual classification. A logical regularization termed L-Reg is derived which bridges a logical analysis framework to image classification. Our work reveals that L-Reg reduces the complexity of the model in terms of the feature distribution and classifier weights. Specifically, we unveil the interpretability brought by L-Reg, as it enables the model to extract the salient features, such as faces to persons, for classification. Theoretical analysis and experiments demonstrate that L-Reg enhances generalization across various scenarios, including multi-domain generalization and generalized category discovery. In complex real-world scenarios where images span unknown classes and unseen domains, L-Reg consistently improves generalization, highlighting its practical efficacy.

Interpret Your Decision: Logical Reasoning Regularization for Generalization in Visual Classification

Abstract

Vision models excel in image classification but struggle to generalize to unseen data, such as classifying images from unseen domains or discovering novel categories. In this paper, we explore the relationship between logical reasoning and deep learning generalization in visual classification. A logical regularization termed L-Reg is derived which bridges a logical analysis framework to image classification. Our work reveals that L-Reg reduces the complexity of the model in terms of the feature distribution and classifier weights. Specifically, we unveil the interpretability brought by L-Reg, as it enables the model to extract the salient features, such as faces to persons, for classification. Theoretical analysis and experiments demonstrate that L-Reg enhances generalization across various scenarios, including multi-domain generalization and generalized category discovery. In complex real-world scenarios where images span unknown classes and unseen domains, L-Reg consistently improves generalization, highlighting its practical efficacy.

1 Introduction

One critical challenge in visual classification models is their ability to generalize effectively to unseen samples or unknown classes. For instance, a model trained on real images of various animals should ideally classify animal sketches accurately (referred to as multi-domain generalization classification [20, 35, 34, 23, 25, 37, 50]) or discover novel categories not present in the training set (referred to as generalized category discovery [54, 16]). These problems are prevalent in realworld scenarios, where training data-

Without L-Reg

With L-Reg

Seen domain

Unseen domain

target pairs are usually insufficient, and labeling is time-consuming so that not every data is paired with a label. Meanwhile, test data is likely to contain shifts in both data and targets, making it essential to propose methods that generalize to border scenarios.

Regularization terms, such as L 2 regularization leading to weight decay, are commonly employed during training to improve a model's generalization capabilities. However, the L 2 regularization

is parametric-based rather than sample-based , which may lead to ambiguous interpretability [58]. As illustrated in Fig. 1, the model trained solely with L 2 regularization exhibits low interpretability. Other regularization terms [57, 58, 59] attempt to improve the interpretability of deep learning models for sequential signals rather than vision, whereas [39] proposes a regularization term to enhance interpretability for robustness in visual classification models rather than generalization. Drawing inspiration from logical reasoning has shown promise for better generalization and interpretability in various tasks. Current work unveils the effectiveness of logical reasoning in generalization tasks, such as boosting performance in length generalization [1, 3, 2, 60] and abstract symbol relational reasoning [10, 36] (e.g., mathematical solving and psychological tests). Several efforts, such as [6], explore the explicit entropy-based logical explanations of neural networks for image classification, confirming the presence and interpretability of logical reasoning within visual tasks. Yet, there are limited studies tackling the generalization of visual classification tasks through the lens of logical reasoning.

This paper studies two pivotal questions corresponding to the above: 1) How does logical reasoning relate to visual tasks such as image classification? 2) How can we derive a logical reasoning-based regularization term to benefit generalization? To achieve these, we correlate the image classification procedure in computer vision with the framework of logic studies [4], positing that training an image classifier involves learning a good general logical relationship between images and labels via an encoder. This good general logic is attained when the semantics generated by the encoder and classifier can be combined to form atomic formulas. Our exploration leads to the introduction of a sample-based Logical regularization term named L-Reg. We reveal that L-Reg efficiently reduces the complexity of the model from two aspects: 1) L-Reg leads to a balanced feature distribution in the semantic space; 2) L-Reg reduces the number of weights with extreme values in the classifier.

Intuitively, the complexity reduction achieved by L-Reg stems from its ability to filter out redundant features or semantics, focusing instead on the minimal yet sufficient semantics for classification defined as semantic support in Definition 3.2, where the interpretability also emerges. This filtering feature benefits the generalization when there is a domain shift in data where the domain-dependent features are ignored for classification. Moreover, it further promotes generalization when unlabeled data from the unknown classes is present. If such data lacks the semantic support associated with known classes, it is then classified as belonging to an unknown class, and its corresponding semantic supports are extracted. These capabilities equip L-Reg with explicit interpretability. As Fig. 1 shows, with L-Reg, the model can identify the unknown class 'person', and pinpoint faces which are the crucial features for classifying this category. In contrast, the model trained solely with L 2 (without L-Reg) focuses on the ambiguous features for classification.

Rigorous theoretical analysis and experimental results validate that L-Reg yields better generalization across diverse scenarios. Specifically, L-Reg facilitates better performance under the aforementioned multi-domain generalization and generalized category discovery tasks, whose settings are presented in Fig. 2 (a)(b). Furthermore, to evaluate L-Reg's robustness, we introduce a more complex real-world scenario, as shown in Fig. 2 (c), where unlabeled images may not only belong to unknown classes, but also originate from unseen domains. Even in this challenging context, L-Reg is still able to consistently demonstrate notable improvements in generalization, underscoring its practical utility and effectiveness. Our code is available at https://github.com/zhaorui-tan/L-Reg_NeurIPS24 .

2 Preliminaries and generalization settings for visual classification

Consider paired ( X,Y ) ∼ ( X , Y ) , ( X s , Y s ) ∼ ( X s , Y s ) , and ( X u , Y u ) ∼ ( X u , Y u ) denote all sets of inputs and labels, seen paired subsets of ( X,Y ) , and unseen paired subsets of ( X,Y ) , respectively. Note that X u , Y u may be accessible for the model separately, but their pairing relationships are not accessible. Let D denote the possible domains, with D s , D u ⊂ D representing the seen and unseen domains. In classification tasks, an encoding function g ( x ) → Z ∈ R M is commonly introduced to map X into the latent feature set Z , where each latent feature has M dimensions. A predictor h ( Z ) → ˆ Y ∈ R K

Sets avaliable for training

Sets unavaliable for training

Sets for evaluation

(a). Multi-domain

generalization

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

Data-shift

(b). Generalized

category discovery

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Target-shift

(c). Multi-domain

generalization + Generalized

category discovery

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

u| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

| GLYPH(cmap:d835)

GLYPH(cmap:d835)

All-shift

maps Z to predictions ˆ Y , where K denotes

Known

Unknown

With L-Reg

Without L-Reg

All classes

With L-Reg

Without L-Reg

Known  classes

Unknown  classes

(b). Distribution of values of classifer's weights under classes

(a). Heatmap of classifer's weights

0.04

Class 0

Class 1

Class 4

Class 5

N

- 0.02

-0.00

Density

1.5

1.0

Density

0.75

0.50

0.25

1.00

0.5

1.0

0.5

1.0

1.0

0.5

1.5

0.02

0.00

Normalized weight value

-1

0

T

0'0

1

Normalized weight value

0

1

0'0

Normalized weight valu

-1

0

0.0

-1

Normalized weight value

0

1

0.5

Class 2

Class 3

Class 6

n

0.04

1.0

1.0

1.5

Aisuaa

1.0

NM

NNN

7

m

26

0.06

0.0+

-1

Normalizedweightvalue

0

1

0.5

0.0 +

Normalized weight value

-1

0

1

0.5

0'0

-1

Normalized weight value

0

1

0.5

0.0

~1

Normalized weight value

0

1

the number of classes and the dimensions of predictions. P ( · ) and H ( · ) symbolize probability and entropy, respectively. This paper discusses two typical cases for generalization in image classification tasks: (1) Data-shift generalization: X s and X u have distribution shifts, such as multi-domain generalization (mDG); and (2) Target-shift generalization: Y s and Y u have distribution shifts, which stands for tasks like generalized category discovery (GCD). We additionally explore a challenging scenario called All-shift generalization: both X s and X u , Y s and Y u have distribution shifts, which is a combination of mDG and GCD tasks (mDG + GCD). The following lists the detailed settings for generalization. Please refer to Fig. 2 for brief diagrams.

̸

Data-shift generalization: Problem setting for mDG. Illustrated in Fig. 2 (a), mDG [9] intends to generalize well to unseen domains having the objective of min H ( X s , Y s | D s ) and expecting the model to be generalized to X u when predicting Y u from the unseen domain D u . In such cases, Y u is fully accessible to the model since Y s and Y u share the same domain: Y s = Y u but there are shifts in X where X s = X u .

̸

Target-shift generalization: Problem setting for GCD. GCD [54] (Fig. 2 (b)) aims to discover possible unseen labels among unlabeled datasets X u . The challenge is that the samples in X u may belong to known classes or unknown classes: Y s = Y u and probably Y s ∩Y u = ∅ . The model should be able to distinguish the samples from the known classes and cluster the samples for unknown classes simultaneously. Note that X u is used for model training, but the relationship between X u and Y u is unseen for the model. In summary, shifts exist between Y s and Y u but not between X s and X u .

̸

All-shift generalization: Problem setting for mDG + GCD. To explore the generalization problem further, we introduce a setting that is the combination of mDG and GCD as shown in Fig. 2 (c). Specifically, the model is trained on the labeled pairs ( X s , Y s ) and unlabeled set X u from the seen domains D s ; X u may belong to known and unknown classes. Furthermore, the model is tested on X u from the unseen domain D u , where X u may also come from the known and unknown classes. In this setting, the model is expected to 1) classify samples to the seen classes and discover the unseen classes among unlabeled samples from seen domains and 2) generalize this ability to the samples from the unseen domain. In this scenario, X s and X u have shifts, and so do Y s and Y u .

For all aforementioned generalization settings, the objective can be summarized as minimizing the generalization loss :

Definition 2.1 (Generalization loss) . Let the target model f ∗ : f ∗ ( X,Y ) : X → Y , can generalize across both seen and unseen sets X,Y . Denote its trainable form f , which is only trained on the seen sets. The generalization loss for the unseen sets is defined as:

3 Logical regularization for generalization in image classification

Under the problem settings defined in Section 2, we introduce Logic regularization (L-Reg) targeting the objective:

where ˆ Y ∈ R K = h ◦ g ( X ) is the prediction set. The corresponding Logic regularization loss (L-Reg) is defined as:

where σ j,i ( ˆ Y T Z ) denotes the value at the i, j position of softmax ( ˆ Y T Z ) and the soft-max function is applied at the last dimension. By incorporating other existing methods' losses denoted by L main , the overall loss is formulated as:

with a weight α applied to balance two losses. As depicted in Fig. 1, L-Reg plays a pivotal role in extracting crucial features for image classification, thus enhancing generalization capabilities. This beneficial outcome can be attributed to two primary factors:

Reducing classifier complexity: L-Reg streamlines the complexity of the classifier itself, as depicted in Fig. 3 (a). Notably, the heat map of the model with L-Reg displays fewer extremely valued weights, evidenced by the diminished presence of intense blue and red colors. This reduction implies that the classifier focuses on leveraging semantically rich and relevant features for decision-making (classification), sidelining the less relevant ones. Additionally, Fig. 3 (b) reveals a reduction in the number of semantic features used to classify each class.

Balancing feature complexity: L-Reg results in a more balanced distribution of features compared to the baseline, as illustrated in Fig. 4. This balanced distribution suggests the elimination of certain extracted semantics characterized by dominant frequencies across all samples. Semantics that occur frequently across samples often lack decisiveness for classification. Hence, reducing their prominence contributes to more expressive feature space and less complex feature distributions. Coupled with the reduced classifier complexity, a simplified classifier achieved through L-Reg facilitates improved generalization across various settings. Specifically, the top row also indicates the distance between the feature distributions of the known and unknown classes, which is enlarged; thus,

Distance between values with most counts of known and unknown classes

Without L-Reg

With L-Reg

they are more dividable, leading to classification improvements.

We present a logical-based theoretical analysis in Section 3.1 and provide the derivation details of L-Reg in Section 3.2. In addition, we discuss the efficacy of L-Reg under various generalization settings in Section 4. Furthermore, L-Reg serves as a plug-and-play loss function that is compatible with most existing frameworks. We conduct experiments applying L-Reg to various established approaches across different generalization settings, as outlined in Section 5.

3.1 Logical framework for visual classification

This part provides the connections between logical reasoning and visual classification tasks. We would like to remind readers of the framework for studying logics and link it with our practical scenarios.

Definition 3.1. Following [4], a logic L is defined as a five-tuple in the form:

where 1) F L denotes the set of formulas formed by images and labels ( X,Y ); 2) M L represents different domains D of X ; 3) | = L is a binary relation relating the truth of whether the formulas are true or false, which has | = L ⊆ M L × F L ; 4) mng L : F L × M L -→ Sets defines the meaning of X as determined by classifiers, where Sets indicate the class of all sets. (5) ⊢ L symbolizes the provability relation of L , evaluating formulas formed by mng L is true or false in one possible world, such as the estimation criteria. More details of L can be seen in Appendix B.

For clarity, we specify L ( X s ,Y s ) = 〈 F ( X s ,Y s ) , D, | = ( X s ,Y s ) , h, ⊢ ( h ( X ) ,Y ) 〉 as the logic formed on the given X,Y sets. With the goal for logic to generalize across a broader scenario and provide extrapolation across all possible formulas in L , a good general logic L ∗ should be derived from L through the feature extractor g :

Importantly, as a good general logic, F ( g ( X s ) ,Y s ) and h in L ∗ should form the atomic formulas , i.e., the tuple of terms with a predicate: h ◦ g ( x ) belongs/not belongs to class y in domain d → Ture/False, where x, y, d ∈ X,Y,D , which makes that ⊢ ( h ◦ g ( X u ) ,Y u ) = | = ( g ( X s ) ,Y s ) still holds. We simply denote one atomic formula in the form of h ( g ( x ) , y, d ) mapping to binary values. Additionally, ⊢ ( h ◦ g ( X ) ,Y ) = | = ( g ( X s ) ,Y s ) in Eq. (6) can be safely omitted in the rest of the paper. Please see more details about the conditions of the good general logic in Appendix B.

An additional tool is necessary to convert the logic problem into a continuous form, enabling the application of machine learning algorithms. The conditional entropy-based method enables a logically sound derivation of knowledge from the provided dataset with constraints [43]. Specifically, the probabilistic inference process adheres to a probabilistic version of Modus Ponens: A → B,A ⊢ B (if A then B ; not A therefore not B ). It is important to note that the logical propositions in probabilistic Modus Ponens are uncertain, with the conditional probability replacing the material implication A → B . This framework allows us to interpret logical deduction through the lens of entropy. Therefore, for Eq. (6) which implies

finding h ◦ g through optimization is equivalent to

where R denotes any other possible regularization.

As the logical framework for image classification takes shape, it becomes evident that the unresolved question of identifying an appropriate function g to generate suitable atomic formulas emerges as a critical factor in ensuring the effectiveness of the overarching logic L ∗ . This paper proposes L-Reg as the regularization to ensure F ( g ( X s ) ,Y s ) are formed by atomic formulas in Section 3.2.

3.2 Constructing atomic formulas using L-Reg

In this part, we show the derivation details of L-Reg the aims to ensure the formation of suitable atomic formulas, as depicted in Eq. (6). As highlighted in [1], current algorithms may induce implicit biases towards unseen data, resulting in varied solutions for such data. However, expecting an algorithm to generalize effectively to unseen data domains without appropriate incentivization, such as specifically designed regularization, is unreasonable. Therefore, we aim to enhance the generalization capability of models by employing a logic-based regularization approach. To this end, we introduce the concept of semantic support for image classification.

Definition 3.2 (Semantic support) . We denote z = g ( x ) , where z ∈ Z , as a set of compositions of these semantics: z := { z i } i M =1 , where M is the number of dimensions or semantics. Notably, not all semantics in z may be useful for deduction or inference. We define the subset γ of z , extracted from the sample x ∼ X , as the semantic support of x if γ is sufficient for deducing the relationship between x and a y ∼ Y .

For instance, if the subset { z 1 , z 2 } ⊆ z is sufficient for accurate inference, the values of other semantics { z i } i M =3 will not impact the inference process. When { z 1 , z 2 } constitutes the minimal combination of semantics required for inference, it is termed the semantic support. We denote Γ as the set of semantic supports of X for deducing each individual class.

Derivation of L-Reg. Regarding Eq. (6), if the semantic supports and their relationship with Y form atomic formulas, Eq. (6) holds as a good general logic, and the generalization would be improved. Thus, we aim to learn the latent features Z , which contain sufficient semantic supports for the deduction of Y :

Specifically, g ( · ) should meet the following:

̸

̸

i.e., the semantic support set for each class should be distinct. The multiple-class classification task has that ∀ Γ , | Γ | ≤ M . Under the constraints demonstrated in Eq. (9) and Eq. (10), we need to achieve the following through optimization:

where ¯ Γ denotes the negation of Γ , i.e., the set of semantics which does not include semantic support.

Intuitively, Eq. (11) regularizes that the model should be able to judge whether a sample belongs to a class by using a minimal set of semantic supports; simultaneously, the semantic support sets are also implicitly disentangled for each class, not only for maintaining rich and useful semantics but also for enhancing the independence of deduction of each class. The actual collection of Γ appears to be intractable during optimization. Hence, we resort to deriving its bounds. Regarding Eq. (11), its former term can be elaborated as follows:

where z i is minimal semantics form z , and E i M =1 H ( Y | g ( z i ) , D ) is the upper-bound for min h,g H ( Y | g (Γ) , D ) . Therefore, minimizing E i M =1 H ( Y | g ( z i ) , D ) is equivalent to minimizing H ( Y | g (Γ) , D ) . Meanwhile, for the latter in Eq. (11), we have:

where H ( Y | h ( z )) , D ) is the lower-bound for max h,g H ( Y | g ( ¯ Γ) , D ) . Combining the aforementioned bounds, we have the L-Reg objective as Eq. (2).

̸

Interpretability of semantic supports roots in forming atomic formulas. The atomic formula A y is of the form h ( g ( x ) , y, d ) . Our aim is to find the good (most) general A y ∗ ∈ A y for y class from which the interpretability of L-Reg is derived. Consider A y 1 , A y 2 ∈ A y , if A y 1 is more general than A y 2 , there will be a substitution ψ such that A y 1 ψ = A y 2 [52]. A y ∗ should meet A y ∗ ψ = A y i ∈ A y , which infers that γ y ψ = z y (cf. Eq. (9)) for predication of y where γ y is the semantic support. Note here that the form of A y is constructed for y ∈ Y , i.e., predicate whether the sample belongs to the y class. Considering multiple classes y i , y j ∈ Y, i = j , it has A y i ∗ = A y j ∗ thus γ y i = γ y j (cf. Eq. (10)), which constrains that different minimal semantic supports should be used for predicting different classes. The interpretability of L-Reg is based on A y ∗ , compelling the model to use distinct minimal semantic supports for each class. These minimal semantic supports can be interpreted as the most critical features for efficient prediction. For example, as shown in Fig. 1, the model with L-Reg has learned the facial features of the person class (see more examples in Appendix Figs. 7 to 12), forming the (informal) atomic formula h ( has a human face , is person , d ∈ D ) → True. Similarly, it also leads to h ( not has a human face , is person , d ∈ D ) → False.

4 L-Reg under different generalization settings

L-Reg under data-shift generalization. The task mDG endeavors to facilitate a model's ability to generalize to unseen domains by fostering invariance across seen domains [50]. In the context of mDG, the term | D | ≥ 2 in Eq. (8) typically denotes multiple domains. Traditionally, existing methods focus on minimizing domain gaps, leading to remarkable results [25, 50]. However, it is noteworthy that even when the domain gap is effectively minimized, and | D | = 1 for the latent features can be considered, L-Reg still demonstrates its efficacy in promoting the generalization of X u from D u .

Proposition 4.1 (Effectiveness of L-Reg in enhancing data-shift generalization.) . Assume the gap across all domains is well minimized. Let f ∗ denote the target model that generalizes to the data X u from the unseen domain with the lowest complexity. For a model f R ( X s ,Y s ) , f ( X s ,Y s ) trained under the data-shift generalization setting (i.e., ( X s , Y s ) is accessible and Y s = Y u ). We have:

Please see proof details in Proposition C.1. To illustrate Proposition 4.1, consider the following intuitive example: In the seen domains, all cats are either black or white, while all dogs are brown. Now, imagine encountering a sample labeled 'a brown cat' from an unseen domain. Without the application of L-Reg, the model might erroneously classify it as a dog. However, with L-Reg in

̸

̸

place, the model is compelled to rely on minimal semantics for classification. This means filtering out irrelevant features such as color terms, thus enabling more accurate deductions.

̸

L-Reg under target-shift generalization. We demonstrate how L-Reg enhances generalized discovery in scenarios where only a subset of classes ( Y s ) is available for training, and there may exist an overlap between the unseen classes ( Y u ) and the seen classes ( Y s ), denoted as Y u ∩Y s = ∅ . We define Y u /Y s as the novel classes not included in Y s , and Y u ∼ Y s as the seen classes for X u classification, where | D | = 1 . Building upon Proposition 4.1, L-Reg further enhances GCG by improving the generalization performance on Y u .

Proposition 4.2 (L-Reg improves target-shift generalization) . When | D | = 1 , L-Reg promotes generalization performance on Y u under the target-shift scenario.

Proof. When | D | = 1 , since all Y belongs to a close set, minimizing -H ( Y s | g ( ¯ Γ) , D ) is equivalent to the following:

where ¯ Y s is the negation of Y s , i.e., Y u /Y s . In this situation, if one sample does not contain sufficient semantic support to be classified under Y s , it otherwise will be assigned under Y/Y s , promoting performance for both Y u /Y s and Y u ∼ Y s . Therefore, the generalization performance on the unseen classes will be improved by L-Reg.

L-Reg under all-shift generalization. When the domain gap is sufficiently minimized and | D | = 1 can be considered, the combination of Proposition 4.1 and Proposition 4.2 demonstrates that L-Reg enhances generalization performance on both novel classes ( Y u /Y s ) and seen classes ( Y u ∼ Y s ) for X u from other domains. Our experiments validate that L-Reg, when applied in scenarios with well-minimized domain gaps, consistently improves generalization across all shifts.

5 Experiments

To validate L-Reg, three groups of experiments under the three kinds of settings are conducted. Notably, all baselines we used already incorporate the L 2 regulation in the form of weight decay. We also compare other commonly used regularization terms, such as independence or sparsity regularization on Z . More results in Appendix F indicate that our L-Reg also surpasses them.

5.1 Experiments on mDG

Experimental settings. We operate on the DomainBed suite [21] and leverage standard leaveone-out cross-validation as the evaluation protocol. We test L-Reg with GMDG [50] on 5 realworld benchmark datasets: PACS [32], VLCS [18], OfficeHome [55], TerraIncognita [7], and

DomainNet [42]. Following MIRO [25] and GMDG [50], the RegNetY-16GF backbone with SWAG pre-training [47]) is used. Specifically, we train the backbone using GMDG with L-Reg. Accuracy is adopted as the evaluation metric, and the results of the averages from three trials of each experiment, with standard deviations, are presented. See Supplementary H for more experimental details.

Results. The experimental results presented in Table 1 demonstrate the efficacy of L-Reg in improving the performance of GMDG across all datasets in mDG classification tasks. Notably, more substantial improvements are observed when the GMDG baseline achieves relatively low accuracy. These observed enhancements provide empirical support for Proposition 4.1. Please see using L-Reg with basic ERM in Appendix E. For detailed insights into each domain within each dataset, please refer to Appendix H.1.

5.2 Experiments on GCD

Experimental settings. We validate our approach through training PIM additionally with L-Reg. Six image datasets are adopted to validate the feasibility of our proposed LReg with PIM compared to other competitors, including three generic object recognition datasets, CIFAR10 [29], CIFAR100 [29] and ImageNet-100 [17]; two fine-grained datasets CUB [56] and Stanford Cars [28]; and the longtail dataset Herbarium19 [49]. Following prior works [54, 16], we use the proposed accuracy metric from [54] of all classes, known classes, and unknown classes for evaluation. Please see a detailed description of the experimental setup in Appendix H.2.

Results. The average results across all datasets for utilizing L-Reg with PIM are presented in Table 2, while detailed dataset-specific information is available in Appendix H Table 17. The results highlight that L-Reg consistently increases the accuracy of all unknown classes across all datasets, thus confirming the validity of Proposition 4.2. However, it is notable that L-Reg may marginally compromise the performance of known classes, as it reduces the size of semantic support for deducing Y , thereby reducing the information available for known classification. Nevertheless, this compromise is deemed acceptable given the significant improvements observed for the unknown classes.

5.3 Experiments on mDG + GCD

Experimental settings. We utilize datasets designed for mDG tasks to conduct mDG + GCD experiments. During the training stage, only samples from seen domains are available, with half of the classes masked as unknown, and only their unlabeled data are utilized. Notably, even though all the unlabeled data originates from unknown classes during training, this prior knowledge is not assumed or constrained, aligning the setting with GCD. Similar to mDG, we adopt the leave-one-out cross-validation method. This entails testing each domain in each dataset as the unseen domain. The performance is tested on unseen domains by employing GCD metrics. To validate L-Reg's efficacy comprehensively, we re-implement four methods under the mDG + GCD setting, testing them both with and without L-Reg. The four methods include ERM, PIM, MIRO, and GMDG. ERM serves as the baseline approach without additional regularization, while PIM maximizes information without minimizing domain gaps. MIRO and GMDG focus on minimizing domain gaps, with GMDG offering a comprehensive approach in this regard. It is worth noting that PIM has been re-implemented. For further experimental details, please refer to Appendix H.3.

Results. The averaged results across all unseen domains of all datasets are summarized in Table 3. For a detailed breakdown of results for each domain in each dataset, please refer to Appendix H.3. As discussed in Proposition 4.1 and Proposition 4.2, a noticeable trend is observed wherein, as the

domain gap is gradually minimized, the improvements for unknown classes increase, with the best results achieved using GMDG with L-Reg. Known class Unknown class

̸

L-Reg forms atomic formulas and improves interpretability. Furthermore, Fig. 5 provides visual insights into the behavior of models trained with LReg. Evidently, these models tend to focus on minimal semantics sufficient for class distinctions. For the known classes, the efficacy of L-Reg can be intuitively understood as extracting the minimal semantic supports for a given class label. For instance, the presence of a guitar's fingerboard, even in unseen domains, helps classify a sample as belonging to the guitar category, whose informal forms can be denoted as h ( has fingerboard , is guitar , d ∈ D ) → True and h ( not has fingerboard , is guitar , d ∈ D ) → False. For all known classes, samples with these minimal semantic supports are recognized accordingly. In contrast, if a sample lacks these minimal supports for any known class, it is very likely categorized as an unknown class. This behavior stems from Paper Eq.10 which ensures A y i ∗ = A y j ∗ through constraining γ y i = γ y j . L-Reg further enhances the model's ability to identify minimal supports for unknown classes

Photo

Sketch

Art painting

cartoon

Seen domains

Useen domain

With L-Reg

Without L-Reg

With L-Reg

Without L-Reg

̸

by filtering out co-covariant features associated with other classes and thus generalizing to unseen domains. Therefore, the very interpretable features for unknown classes from unseen domains can be extracted using L-Reg. Fig. 5 (right side) demonstrates that the model with L-Reg can even extract facial features for the unknown person class and can generalize this to the unseen domain. Similarly, here we obtain (informal) atomic formulas as h ( has a face , is person , d ∈ D ) → True, h ( not has a face , is person , d ∈ D ) → False.

However, as shown in Row 3, significant domain shifts, such as those between the sketch domain and other domains, pose challenges. Specifically, the differences between the stick-figure style of sketches of persons and figures from other domains can hinder the model's ability to cluster sketches with other domains' figures when the class label is unknown. Thus, under this circumstance, the model may fail to extract meaningful features from those sketches. We acknowledge this limitation and will explore solutions in future work.

L-Reg should be applied to features from deep layers. One crucial precondition highlighted in the theoretical analysis is that L-Reg operates effectively with a representation Z , where each dimension represents independent semantics. The semantic features usually come from the deeper layers of the model

architecture [51]. However, Table 4 shows that applying L-Reg to features from earlier layers, which may not necessarily represent semantics, leads to a degradation in performance for known classes, albeit improving performance for unknown classes. This phenomenon arises due to the potential interdependence among features from earlier layers, resulting in penalization that may hinder the capture of semantic supports essential for known classes. To ensure generalization improvements without significant compromise to the performance of known classes, we advocate for applying L-Reg specifically to features extracted from deeper layers, such as the bottleneck layer. These suggest that the compromised results observed in Table 2 could be attributed to the less depth of the model structure, which fails to provide the expected semantic features.

5.4 Apply L-Reg to congestion prediction for circuit design.

Experimental settings. We also test L-Reg in Congestion prediction on the CircuitNet [15] dataset by using CircuitFormer [63] backbone. The congestion prediction is for circuit design and benefits from logical reasoning-based approaches. All parameters, except for L-Reg, remain consistent with CircuitFormer, and we follow its metrics.

Results. Table 5 shows the results of prediction results on the CircuitNet dataset. We also include the results of Gpdl with UNet++ and CircuitFormer for better comparison. Notably, the improvements brought by CircuitFormer with LReg across all metrics, especially for the pearson metric can be observed. The consistent improve-

ment with L-Reg across all metrics indicates L-Reg's feasibility.

6 Related work

Logical reasoning for deep learning. Current studies focus on length generalization or symbolic reasoning in the logic-based scope. For length generalization, [1] proposes the generalization to the unseen setting, theoretically verifying that commonly used models can generalize to the unseen and degree curriculum promotes the generalization ability of the transformer, followed by [3, 2, 60]. Another branch is to improve the logical reasoning ability for abstract symbols, such as learning the logical-based temples and expecting the model to generalize to unseen samples [10, 36]. These studies are closely related to languages, such as generating longer answering sequences or solving mathematical problems in large language models, lacking explicit connections to visual tasks. [6] delves into the logical explanations in image classification by explicitly extracting logical relationships. While this logical-based approach sheds light on the interpretability of image classification models, its specific benefits for visual generalization remain relatively unexplored.

Multi-domain generalization. Current approaches for mDG in image classification focus on learning invariant representation across domains. Previous approaches like DANN [20] minimize feature divergences between source domains. CDANN [35], CIDG [34], and MDA [23] consider conditions for learning conditionally invariant features. MIRO [25] and GMDG [50] take advantage of pretrained models to improve generalization. Specifically, in comparison to MIRO, GMDG proposes a general entropy-based learning objective for mDG and sufficiently minimizes the domain gaps, yielding better generalization results.

Generalized category discovery. Generalized category discovery, pioneered by [54], addresses unlabeled samples with both known and unknown classes. Furthermore, PIM [16] integrates InfoMax into generalized category discovery, effectively handling imbalanced datasets and surpassing GCD on both short- and long-tailed datasets.

7 Conclusion

This paper presents L-Reg, a logical regularization approach tailored for image classification tasks using logic analysis frameworks. L-Reg yields better generalization across different settings by fostering balanced feature distributions and streamlining the classification model's complexity. Rigorous theoretical analyses and empirical validations underscore its efficacy, as L-reg consistently improves generalization performance with different frameworks under various scenarios.

Limitation. L-Reg narrows the extent of semantic supports, potentially diminishing the amount of information available for classification and leading to certain trade-offs in the performance of seen datasets. This effect is evidenced by the slight decline in the accuracy of known classes when L-Reg is applied, as shown in Table 2. A similar phenomenon is observed in Fig. 5, where the model fails to recognize a person in the sketch domain lacking facial features. Analysis from Table 4 suggests that these compromises may result from improper Z . Future work should focus on mitigating potential compromises on seen datasets by exploring strategies for better capturing Z through improved model architecture design. We offer more experimental results of possible solutions to this limitation in Appendix G, such as further constraining the independence of each dimension in Z . Those results may suggest a direction for future work.

Acknowledgments

The work was partially supported by the following: National Natural Science Foundation of China under No. 92370119, No. 62376113, No. 62206225, and No. 62276258.

References

Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk. Generalization on the unseen, logic reasoning and degree curriculum. In International Conference on Machine Learning , pages 31-60. PMLR, 2023.

Emmanuel Abbe, Elisabetta Cornacchia, and Aryo Lotfi. Provable advantage of curriculum learning on parity targets with mixed inputs. Advances in Neural Information Processing Systems , 36, 2024.

Kartik Ahuja and Amin Mansouri. On provable length and compositional generalization. arXiv preprint arXiv:2402.04875 , 2024.

Hajnal Andréka, István Németi, and Ildikó Sain. Universal algebraic logic. Studies in Logic, Springer, due to , 2017.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, and Stefano Melacci. Entropy-based logic explanations of neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6046-6054, 2022.

Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV) , pages 456-473, 2018.

Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research , 22(1):46-100, 2021.

Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems , 24, 2011.

Enric Boix-Adsera, Omid Saremi, Emmanuel Abbe, Samy Bengio, Etai Littwin, and Joshua Susskind. When can transformers reason with abstract symbols? arXiv preprint arXiv:2310.09753 , 2023.

Malik Boudiaf, Imtiaz Ziko, Jérôme Rony, José Dolz, Pablo Piantanida, and Ismail Ben Ayed. Information maximization for few-shot learning. Advances in Neural Information Processing Systems , 33:2445-2457, 2020.

Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specific features to enhance domain generalization. Advances in Neural Information Processing Systems , 34:21189-21201, 2021.

Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In International Conference on Learning Representations , 2022.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Zhuomin Chai, Yuxiang Zhao, Wei Liu, Yibo Lin, Runsheng Wang, and Ru Huang. Circuitnet: An open-source dataset for machine learning in vlsi cad applications with improved domain-specific evaluation metric and learning strategies. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 42(12):5034-5047, 2023.

Florent Chiaroni, Jose Dolz, Ziko Imtiaz Masud, Amar Mitiche, and Ismail Ben Ayed. Parametric information maximization for generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1729-1739, 2023.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision , pages 1657-1664, 2013.

Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9284-9292, 2021.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research , 17(1):2096-2030, 2016.

Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434 , 2020.

Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. In Uncertainty in Artificial Intelligence , pages 292-302. PMLR, 2020.

Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 124-140. Springer, 2020.

Cha Junbum, Lee Kyungjae, Park Sungrae, and Chun Sanghyuk. Domain generalization by mutualinformation regularization with pre-trained models. European Conference on Computer Vision (ECCV) , 2022.

Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9619-9628, 2021.

Andreas Krause, Pietro Perona, and Ryan Gomes. Discriminative clustering by regularized information maximization. Advances in neural information processing systems , 23, 2010.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops , pages 554-561, 2013.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning , pages 5815-5826. PMLR, 2021.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision , pages 5542-5550, 2017.

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5400-5409, 2018.

Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European conference on computer vision (ECCV) , pages 624-639, 2018.

Zenan Li, Yunpeng Huang, Zhaoyu Li, Yuan Yao, Jingwei Xu, Taolue Chen, Xiaoxing Ma, and Jian Lu. Neuro-symbolic learning yielding logical constraints. Advances in Neural Information Processing Systems , 36, 2024.

Ziyue Li, Kan Ren, Xinyang Jiang, Yifei Shen, Haipeng Zhang, and Dongsheng Li. Simple: Specialized model-sample matching for domain generalization. In The Eleventh International Conference on Learning Representations , 2022.

J MacQueen. Classification and analysis of multivariate observations. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability , pages 281-297, 1967.

Ofir Moshe, Gil Fidel, Ron Bitton, and Asaf Shabtai. Improving interpretability via regularization of neural activation sensitivity. arXiv preprint arXiv:2211.08686 , 2022.

Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8690-8699, 2021.

Konstantinos Panagiotis Panousis, Dino Ienco, and Diego Marcos. Sparse linear concept discovery models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2767-2771, 2023.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1406-1415, 2019.

Wilhelm Rödder. Conditional logic and the principle of entropy. Artificial Intelligence , 117(1):83-106, 2000.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937 , 2021.

Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 804-814, 2022.

Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14 , pages 443-450. Springer, 2016.

Kiat Chuan Tan, Yulong Liu, Barbara Ambrose, Melissa Tulig, and Serge Belongie. The herbarium challenge 2019 dataset. arXiv preprint arXiv:1906.05372 , 2019.

Zhaorui Tan, Xi Yang, and Kaizhu Huang. Rethinking multi-domain generalization with a general learning objective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 23512-23522, June 2024.

Zhaorui Tan, Xi Yang, and Kaizhu Huang. Semantic-aware data augmentation for text-to-image synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 5098-5107, 2024.

Irene Tsapara and György Turán. Learning atomic formulas with prescribed properties. In Proceedings of the eleventh annual conference on Computational learning theory , pages 166-174, 1998.

Vladimir N. Vapnik. Statistical Learning Theory . Wiley-Interscience, 1998.

Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7492-7501, 2022.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5018-5027, 2017.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Chunyang Wu, Mark JF Gales, Anton Ragni, Penny Karanasou, and Khe Chai Sim. Improving interpretability and regularization in deep learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 26(2):256-265, 2017.

Mike Wu, Michael Hughes, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi-Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Mike Wu, Sonali Parbhoo, Michael Hughes, Ryan Kindle, Leo Celi, Maurizio Zazzi, Volker Roth, and Finale Doshi-Velez. Regional tree regularization for interpretability in deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 6413-6421, 2020.

Changnan Xiao and Bing Liu. A theory for length generalization in learning to reason. arXiv preprint arXiv:2404.00560 , 2024.

Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. Advances in Neural Information Processing Systems , 34:23664-23678, 2021.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint arXiv:2104.02008 , 2021.

Jialv Zou, Xinggang Wang, Jiahao Guo, Wenyu Liu, Qian Zhang, and Chang Huang. Circuit as set of points. Advances in Neural Information Processing Systems , 36, 2024.

A Broader impact

Our regularization term based on logic for image classification offers significant potential beyond academia. By integrating logical constraints, our approach enhances model robustness, interpretability, and ethical alignment. This translates into improved performance on real-world tasks such as disease diagnosis in healthcare and mitigating biases in decision-making systems. Our work fosters interdisciplinary collaboration and contributes to the responsible deployment of AI technologies, ultimately benefiting society through enhanced efficiency, fairness, and transparency in machine learning applications.

B Details of the logical framework for visual classification task

We provide more details of the connections between logical reasoning and visual classification tasks. Definition B.1. Following [4], a logic L is a five-tuple defined in the form:

F L is a set of all formulas of L . F L arbitrarily refers to any collections that can be 'expressed' by language L . Therefore, F L could be not only a collection of languages but also images and labels ( X,Y ) for computer vision cases.

M L is a class called the class of all models (or possible worlds) of L ; intuitively, this can be considered as different domains D of X .

| = L is a binary relation, | = L ⊆ M L × F L , called the validity relation of L . For example, in the known set, the ground truth label of the image is given as truth, which is the validity relation.

mng L : F L × M L -→ Sets where Sets is the class of all sets. mng L is a function with domain F L × M L , called the meaning function of L . Intuitively, mng L extracts the meaning of the expressions that can be understood as the classifiers.

⊢ L represents the provability relation of L , telling us which formulas are 'true' in which possible world and usually is definable from mng L , such as the estimation criteria in the machine learning system.

Accordingly and still following [4], a good general logic is defined as:

Definition B.2 (General logic) . : A general logic is a class:

where Sig is a class of sets; L P = 〈 F P L , M P L , | = P L , mng P L , ⊢ P L , 〉 is a compositional logic in the sense of Definition B.1 for P ∈ Sig , and for any sets P, Q ∈ Sig satisfies the following conditions:

P is the set of atomic formulas of L P .

Cn ( L P ) = Cn ( L Q ) := Cn ( L ∗ ) where Cn ( · ) is called the set of logical connectives of the given logic (these are operation symbols with finite or infinite ranks).

Any bijection f : P ß Q that extends to a bijection between the tautological formula algebras of L P and L Q induces an isomorphism between L P and L Q .

If P ⊆ Q , then L P is a sublogic of L Q .

For any P ∈ Sig and set H , there is a P ′ ∈ Sig such that P ′ is disjoint from H and L P ′ is an isomorphic copy of L P .

The union of a system P i , i ∈ I of pairwise disjoint sets P i from Sig belongs to Sig , whenever I is not empty. Let Fr ( · ) denotes free algebra, Alg m ( L ) represents { mng M ( F ) : M ∈ M } where F denotes the term algebra. Further, the tautological congruence of the logic belonging to the disjoint union P is generated in Fr ( Alg m ( L P ) , P ) as a

where

congruence by the union of the tautological congruence relations of the logics belonging to P i , i ∈ I .

7. Sig contains at least one non-empty set.

Our L-Reg aims to regularize the semantics extracted by g and the classifier to satisfy condition 1 .

⊢ ( h ◦ g ( X ) ,Y ) = | = ( g ( X s ) ,Y s ) in Eq. (6) can be safely omitted in the rest of the paper. Consider the logic formed on X,Y : L ( X s ,Y s ) = 〈 F ( X s ,Y s ) , D, | = ( X s ,Y s ) , h, ⊢ ( h ( X ) ,Y ) 〉 . Assume we want to study the logic of ⊢ which can be defined in the form of L ⊢ : def = ⟨ F X s ,Y s , D ⊢ , h ⊢ , | = ⊢ ⟩ , where D ⊢ , h ⊢ , | = ⊢ are pseudo-components associated with ⊢ . Particularly, D ⊢ is a subset of all possible world/domains from F ( X s ,Y s ) : D ⊢ def = { T ⊆ F ( X s ,Y s ) : T is closed under ⊢ ( h ( X ) ,Y ) } . For any T ∈ D ⊢ and a ∈ F ( X s ,Y s ) , it has h ⊢ ( a, T ) def = { b ∈ F : T ⊢ ( a ↔ b ) } . Further, | = ⊢ in T ∈ D ⊢ is defined as T | = ⊢ a def ⇔ a ∈ T . [4] points out that the following condition is almost always satisfied: (Cond) ∀ a, b ∈ F ⊢ , d ∈ D ⊢ , we have ( h ⊢ ( a, d ) = h ⊢ ( b, d )) and d | = ⊢ a ⇒ d | = ⊢ b . Therefore, the semantical consequence relation induced by | = ⊢ coincides with the original syntactical ⊢ ( h ◦ g ( X ) ,Y ) while Cond holds. Due to that D ⊢ ⊆ D , it infers that | = ( g ( X s ) ,Y s ) coincides with | = ⊢ . Therefore, ⊢ ( h ◦ g ( X ) ,Y ) = | = ( g ( X s ) ,Y s ) can be safely omitted in the rest of the paper.

C Details of proofs

Proposition C.1 (L-Reg reduces the complexity of the model, promoting data-shift generalization performance.) . Assume the domain gap is well minimized. Consider a f ∗ is the target model that generalizes to the unseen with the lowest complexity. There are f R ( X s ,Y s ) , f ( X s ,Y s ) trained under the setting of data-shift generalization (i.e., ( X s , Y s ) is accessible and Y s = Y u ), it has that:

Proof. We assume the loss is achieved for the tractable form by minimizing the mean squared error. In that case, we have f ∗ ( X s ,Y s ) for the given training set as:

In comparison to f ∗ , f ( X s ,Y s ) for the given seen sets is as:

and f R ( X s ,Y s ) is derived from f ( X s ,Y s ) , where Z s is constrained additionally by L-Reg and the constrained Z s is denoted as Z R s :

For simplification, we denote ( Z T Z ) -1 Z T , ( Z T s Z s ) -1 Z T s , and ( Z R T s Z s ) -1 Z R T s as N ∗ , N , and N R , respectively.

̸

The form of N . For multi-domain generalization, the model is tested on the unseen domain, referring that X u contains some unseen semantics besides the seen: Z s ∼ Z s , Z u ∼ Z u , Z s = Z u , Z s ∩Z u = ∅ . Considering each dimension of Z represents a specific semantics, we denote Γ as the dimensions of Z that contain the seen semantics support in X s and ¯ Γ for the unseen, we can decompose N as:

̸

The form of N ∗ . Assume Γ already contains semantic support for deducting Y ; thus, ¯ Γ would not affect the deduction of Y . In such case, it has that Γ T ¯ Γ = 0 and ¯ Γ T Γ = 0 and ¯ Γ T ¯ Γ = 1 where 0 , 1

denote zero matrix and identity matrix:

where we also expect h ( ¯ Γ) = 0 so that z u does not influence the deduction. We now have N ∗ :

Note that for N in f ( X s ,Y s ) , Γ T ¯ Γ and ¯ Γ T Γ are not constrained. Please refer to Lemma C.2. Furthermore, h ( ¯ Γ) is also not constrained.

The form of N R . Now we discuss the trainable N R obtained with the application of L-Reg. The form of N R is similar to N ∗ . However, Eq. (11) indicates that L-Reg minimizes || Γ T ¯ Γ || 2 and || ¯ Γ T Γ || 2 through -H ( Y | g ( ¯ Γ)) , D ) and also minimizing || h ( ¯ Γ) || 2 :

Compare GL ( f R ( X s ,Y s ) , f ∗ , X u ) with GL ( f ( X s ,Y s ) , f ∗ , X u ) . By comparing the forms of N R , N and N ∗ , it is obvious that ||N R - N ∗ || 2 ≤ ||N - N ∗ || 2 . Therefore, we have that: GL ( f R ( X s ,Y s ) , f ∗ , X u ) ≤ GL ( f ( X s ,Y s ) , f ∗ , X u ) .

Lemma C.2 (Minimizing H ( Y | g ( X ) , D ) + R solely may cause generalization degradation) . Minimizing H ( Y | g ( X ) , D ) + R solely without L-Reg may conflict with max h,g H ( Y | g ( ¯ Γ) , D ) , causing invalid semantics for decision process and degrading the generalization.

Proof. We have the following relationship for H ( Y | g ( z ) , D ) :

Since the independence between { z i } i M =1 is unconstrained, H ( Y, g (Γ) | g ( ¯ Γ) , D ) may cause that Y can be deducted from ¯ Γ . Therefore, Γ T ¯ Γ and ¯ Γ T Γ are not constrained even when the domain gap is minimized where | D | = 1 , causing the sub-optimal generalization.

D One toy example

We present a simplified informal illustrative example to compare the efficacy of our proposed L-Reg against conventional L1 and L2 regularization methods. As depicted in Fig. 6, the ground truth (GT) image represents the underlying data, generated according to f ∗ ( x 1 , x 2 ) = sin(2 πx 1 ) · sin(2 πx 2 ) , where x 1 and x 2 denote the horizontal and vertical coordinates respectively, and the pixel color corresponds to the value of f ∗ ( x 1 , x 2 ) . The training domain is delineated by the black box, while the testing domain encompasses the area outside of this boundary.

For our experiments, we use a 6-linear-layer size-110 ReLU model network. Mean squared error serves as the loss function.

Our experimental results reveal that L-Reg enhances the model's ability to extrapolate beyond the training domain. Notably, our proposed L-Reg demonstrates superior extrapolative capabilities compared to traditional L 1 and L 2 regularization methods. This observation highlights the efficacy of L-Reg in fostering improved generalization.

Base model

+

GLYPH(cmap:d835)1

Regularization

+

GLYPH(cmap:d835)2

Regularization

+ L-Reg

GT

E Apply L-Reg to ERM Baseline for mDG

To further validate L-Reg's efficacy for mDG, we use ERM as the baseline on the TerraIncognita dataset. For a fair comparison, all experiments share the same hyperparameter settings and use the Regnety-16gf backbone. Original ERM results are also included alongside our reproduced results. The results in Table 8 reveal that ERM with L-Reg significantly improves mDG performance (from 49.9% to 52.9%).

F Compare L-Reg with more regularization terms

We also compare L-Reg with other regularization terms: The Ortho-Reg - the orthogonality regularization that constrains the independence of each dimension of the semantic feature z ; and Sparsity - implemented as Bernoulli Sample of the latent features from the sparse linear concept discovery models [41] on our used PIM backbone. To investigate this fairly, we re-implemented the Bernoulli Sample of the latent features from the Sparse Linear Concept Discovery Models [41] on the same PIM backbone that we used, to achieve the sparsity. Table 6&Table 7 demonstrate that L-Reg outperforms Ortho-Reg and Sparsity.

̸

Especially, while a common sparse concept model may be able to achieve γ y ψ = z y by filtering irrelevant features through the sparsity, it may not ensure γ y i = γ y j , which is crucial for disentangling features used for predicting different classes. This limitation can potentially lead to degradation in generalization performance for common sparse concept models. 6&7 indicate that while L-Reg consistently achieves overall improvement, the sparse concept-based approach does not consistently improve generalization, validating the aforementioned difference.

G Limitation of L-Reg and possible solutions

As analyzed and discussed in the paper, L-Reg is based on the precondition that each dimension of the latent features represents an independent semantic.

̸

We hypothesize this is due to the fact that our L-Reg is derived based on the precondition that z i , z j ∈ z, I = j is independent of each other. This condition holds for most deep-layer features but may not apply to shallow layers. Thus, applying L-Reg to the semantic features from the deep layers may improve the performance for unknown classes without negatively impacting known classes.

Derived from this hypothesis, another possible solution is further regularizing the independence, which may lead to further improvements. To validate this hypothesis, we test L-Reg by reinforcing independence with Ortho-Reg. MDG results in Table 8 and GCD results in Table 6&Table 7 show that combining L-Reg with Ortho-Reg leads to further improvements, whereas Ortho-Reg alone may not guarantee improvements. These findings support our hypothesis and suggest that L-Reg, particularly

when applied to deep layers or in conjunction with Ortho-Reg, is beneficial. This suggests a direction for future work.

H More experimental details and results

All experiments can be conducted on one NVIDIA GeForce RTX 3090 GPU.

H.1 Multi-domain generalization

Competitors. We listed results from previous important work in the mDG field for better validation. They are: MMD [33], Mixstyle [62], GroupDRO [44], IRM [5], ARM [61], VREx [30], CDANN [35], DANN [20], RSC [24], MTL [8], MLDG [31], Fish [46], ERM [53], SagNet [40], SelfReg [26], CORAL [48], mDSDI [12], MIRO [25], and GMDG [50]. Among them, GMDG is treated as our baseline since it sufficiently minimizes the domain gaps.

Datasets. We use PACS (4 domains, 9,991 samples, 7 classes) [32], VLCS ( 4 domains, 10 , 729 samples, 5 classes) [18], OfficeHome (4 domains, 15,588 samples, 65 classes) [55], TerraIncognita

(TerraIncognita, 4 domains, 24 , 778 samples, 10 classes) [7], and DomainNet (6 domains, 586,575 samples, 345 classes) [42].

Training details. We use GMDG [50] as our baseline. Especially, we use all loss terms proposed in GMDG as L main . The training procedure is the same as MIRO [25] and GMDG. We use seeds 0 , 1 , 2 for all three trails training.

Parameters. We adhere to the parameters proposed by GMDG, particularly focusing on its recommended loss terms. Furthermore, we provide a detailed listing of the hyper-parameters pertaining to L-Reg, along with the tuned 'lr mult', as outlined in Table 9, to facilitate the reproducibility of our results.

Evaluation metric. The models undergo training on known domains and subsequent testing on unseen domains. For each trial, a distinct domain within the datasets is designated as the unseen domain. The evaluation metric reports the prediction accuracy achieved on these unseen domains. The aggregated results across all unseen domains within the datasets provide a comprehensive assessment of the algorithm's performance in domain generalization for the given datasets.

More results. Results of each domain for each dataset are presented in Tables 10 to 14.

H.2 Generalized category discovery

Competitors. We compare our proposed method with existing generalized category discovery methods: GCD [54], and PIM [16]. In particular, PIM based on information maximization is the current state-of-the-art (SOTA) generalized category discovery method. Additionally, the traditional machine learning method, k-means [38]; three novel category discovery methods: RankStats+ [22], UNO+ [19], ORCA [13]; and several information maximization methods: RIM [27], and TIM [11] are adapted for generalized category discovery as competitors. The results of the modified novel category discovery methods are reported in [54], and the modified information maximization methods are reported in [16].

Usage details of datasets for GCD. Following the protocols of GCD and PIM [54, 16], the initial training set of each dataset is divided into labeled and unlabeled subsets; samples from half of the classes are assigned as unlabeled, and their labels are not used for training. Specifically, half of the image samples from known classes are allocated to the labeled subset, while the remaining half are assigned to the unlabeled subset. Additionally, the unlabeled subset includes all image samples from

the novel classes in the original dataset. As a result, the unlabeled subset consists of instances from K different classes. The detailed statistics of datasets are listed in Table 15.

Training details. Consistent with PIM, we utilize latent features extracted by the feature encoder DINO (VIT-B/16) [14] that is pre-trained on ImageNet [17] through self-supervised learning. The losses proposed in PIM are treated as L main . The original PIM freezes the feature extractor during the training, directly using the pre-saved extracted features as the model input. For a fair comparison, we only added one linear layer as g on the extracted features, which is the minimal modification.

L 2 (weight decay) value searching. For a more fair comparison, we conduct weight decay value searching to ensure that the weight of L 2 is the best. To address this, we devised a methodology for weight decay searching involving the construction of smaller labeled and unlabeled subsets derived solely from the labeled data. To conduct parameter searching, we split the labeled samples to construct a 'smaller' sub-labeled and sub-unlabeled set. Specifically, we take 50% of the samples from known classes as sub-unlabeled samples from unknown classes. Additionally, we take 25% of the samples from the remaining 50% of known classes as sub-unlabeled samples from known classes. The remaining samples are treated as sub-labeled samples. Hyper-parameters are then searched on these sub-labeled and sub-unlabeled sets.

Parameters of L-Reg. The hyper-parameters of L-Reg values are shown in Table 16.

Evaluation metric. Following prior works [54, 16], we use the proposed accuracy metric from [54] of all classes, known classes, and unknown classes for evaluation.

More results. The results for each dataset are presented in Table 17. It is evident that L-Reg yields enhanced performance across half of the datasets for both known and unknown classes. On the remaining datasets, while L-Reg may slightly compromise the performance of known classes, it demonstrates significant improvements in the unknown classes, resulting in an overall enhancement in the performance across all classes.

More ablation results. Due to the introduction of tuned weight decay and the additional g component, we have conducted ablation studies to assess their impact. The results are summarized in Table 18. It is observed that the baseline model utilizing the tuned weight decays performs slightly better than the original weight decay settings. Notably, the tuned weight decays contribute to improvements in unknown classes while often leading to slight decreases in known classes across most datasets. Inclusion of the proposed extra component g results in marginal improvements in both known and unknown classes compared to the tuned baseline. Our proposed L-Reg demonstrates significant improvements specifically in the unknown classes, thereby corroborating Proposition 4.2. However, as discussed in the main paper, it is acknowledged that L-Reg may entail compromises in the performance of known classes.

H.3 Combination of multi-domain generalization and generalized category discovery

Datasets. We leverage the datasets utilized in mDG tasks to construct the mDG+GCD datasets. Specifically, during the seen domains of training, labels from approximately half of the classes are masked. For instance, in the PCAS dataset comprising 7 classes, classes labeled within the range [0 , 1 , 2 , 3] are retained, while classes in [4 , 5 , 6] are masked. It is noteworthy that data categorized as unknown classes in our setup are from unknown classes. However, we acknowledge that this prior is not explicitly known. To align with the GCD setting, we operate under the assumption that the unlabeled set may potentially include samples from known classes. Consequently, we refrain from constraining the model by mandating that unlabeled data be classified solely as unknown classes. This adjustment introduces a more challenging generalization scenario.

Training details. For all experiments, the implementation directly adds L-Reg to their previously proposed loss sets. The models are trained with the aforementioned labeled and unlabeled sets from the seen domains and tested on the samples from the unseen domain.

Parameters. We include all the parameters for reproducing our experiments in the code. Please refer to the code for details.

Evaluation metric. We use the same metric from the GCD task for the mDG+GCD task. Similarly, the metrics include the accuracy for all, known and unknown classes.

More results. The averaged results of each dataset are exhibited in Table 19, while the detailed results of each dataset are presented in Tables 20 to 24.

H.4 More GradCAM visualizations

We provide more visualized examples of L-Reg. Examples of known classes can be seen in Figs. 7 to 10 and unknown classes in Figs. 11 and 12. Compromises in known sets, as discussed in the limitations, can be seen in Figs. 8 and 12.

Unseen domain: Art painting

Without

L-Reg

With

L-Reg

Seen domain: Photo

Without

L-Reg

With

L-Reg

Without

L-Reg

With

L-Reg

Seen domain: Sketch

Without

L-Reg

With

L-Reg

Seen domain: Cartoon

Unseen domain: Art painting

Without

L-Reg

With

L-Reg

Seen domain: Photo

Without

L-Reg

With

L-Reg

Without

L-Reg

With

L-Reg

Seen domain: Sketch

Without

L-Reg

With

L-Reg

Seen domain: Cartoon

8

?

Unseen domain: Art painting

Without

L-Reg

With

L-Reg

Seen domain: Photo

Without

L-Reg

With

L-Reg

Without

L-Reg

With

L-Reg

Seen domain: Sketch

Without

L-Reg

With

L-Reg

Seen domain: Cartoon

shutterst

Unseen domain: Art painting

Without

L-Reg

With

L-Reg

Seen domain: Photo

Without

L-Reg

With

L-Reg

Without

L-Reg

With

L-Reg

Seen domain: Sketch

Without

L-Reg

With

L-Reg

Seen domain: Cartoon

Store

StoreNo

Unseen domain: Art painting

Without

L-Reg

With

L-Reg

Seen domain: Photo

Without

L-Reg

With

L-Reg

Without

L-Reg

With

L-Reg

Seen domain: Sketch

Without

L-Reg

With

L-Reg

Seen domain: Cartoon

Unseen domain: Art painting

Without

L-Reg

With

L-Reg

Seen domain: Photo

Without

L-Reg

With

L-Reg

Without

L-Reg

With

L-Reg

Seen domain: Sketch

Without

L-Reg

With

L-Reg

Seen domain: Cartoon

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, the paper discusses the limitations of the work in the Conclusion Section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, the paper provides the full set of assumptions and a complete proof in the main paper and appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper provides all the information needed to reproduce the main experimental results of the paper as much as possible in the appendix and supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provides uses open-source data and open access to the code. The code is especially attached as supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, this paper specifies all the training and test details in the Experiments Section and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Following previous work, this paper does not report error bars. However, this paper reports some results in the form of mean with standard deviation, as shown in Table 1. However,

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides information on the computer resources in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We think that there is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper seems to pose no such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All assets (e.g., code, data, models) used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: New assets introduced in the paper are documented, and the documentation is provided alongside the assets. The model training code is provided in the supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a logical regularization method called L-Reg, aimed at enhancing the generalization ability of image classification tasks through a logical reasoning framework. L-Reg effectively simplifies the complexity of the model by ensuring that the generated atomic formulas align with the logical relationships between the images and labels, promoting balanced distribution in the feature space, and reducing the number of extreme weight values in the classifier. Both theoretical analysis and experimental results validate the effectiveness of L-Reg in various generalization scenarios, especially demonstrating outstanding performance in multi-domain generalization and generalized category discovery tasks.\n\nSTRENGTHS:\n1. L-Reg effectively reduces the complexity of the model by balancing the feature distribution and reducing the number of extreme weight values in the classifier.\n\n2. In Sections 3.1 and 3.2, the paper provides logic-based theoretical analysis and detailed derivation of the construction process of L-Reg. Furthermore, through experiments in Sections 4 and 5, the effectiveness of L-Reg in different generalization settings, especially in multi-domain generalization and generalized category discovery tasks, is validated.\n\n3. Designed as a plug-in loss function, L-Reg is compatible with most existing frameworks, making it highly flexible and practical in real-world applications.\n\n4. The paper also explores the relationship between logical reasoning and visual classification tasks, delving into the derivation of logic-based regularization terms to promote generalization, providing new perspectives and methods for research in related fields.\n\nWEAKNESSES:\n1. L-Reg may reduce the scope of semantic support, leading to a slight performance decrease on known datasets. It is hoped that the authors can analyze in more detail the reasons for this performance drop and provide possible improvement methods.\n\n2. The appropriate function g for generating atomic formulas is a key factor. While the authors propose L-Reg as a regularization method to ensure that F(g(Xs), Ys) consists of atomic formulas, they do not elaborate on how to choose or design this function g. Further explanation on the specific methods or criteria for selecting g is desired.\n\n3. In multi-domain generalization tasks, how L-Reg integrates with existing methods and how to adjust alpha to balance the two losses are details that the authors are encouraged to further elucidate.\n\nQUESTIONS:\nI hope the authors can further compare L-Reg with other regularization methods in terms of generalization ability and interpretability to highlight the advantages of L-Reg.\n\nLIMITATIONS:\nAuthors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the multi-domain generalization (mDG), generalized category discovery (GCD), and the more challenging mDG+GCD task. The authors introduce a logical reasoning-based regularization term called L-Reg, which bridges logical analysis with image classification to enhance model interpretability and reduce complexity. The main idea of L-Reg is to identify a minimal set of semantics that can deduce the relationship between $x$ and $y$, i.e., the semantic support. Theoretical analysis and experiments demonstrate that L-Reg improves generalization across mDG, GCD, and mDG+GCD.\n\nSTRENGTHS:\n1. This paper establishes a connection between logical reasoning and practical visual generalization problems, bringing novel insights for improving DG and GCD.\n2. The proposed L-Reg used over existing SOTA methods can further improve the SOTA performance.\n3. The experiments are comprehensive. The visualization results provide a good empirical understanding of the role of L-Reg.\n\nWEAKNESSES:\n1. The presentation of this article is somewhat confusing for readers unfamiliar with logical reasoning, especially from Section 3.1 to Section 3.2. It would be better to add intuitive explanations of key concepts. For example, why should $F$  be atomic formulas for a logic to be a 'good general' one, or, what do the atomic formulas refer to in a real-world case?\n2. The mDG improvements of L-Reg are over GMDG. The performance of directly applying L-Reg is not shown, i.e., ERM+L-Reg.\n\nQUESTIONS:\n1. Line 145 claims that \"Semantics that occur frequently across samples often lack decisiveness for classification.\" However, samples from the same class often share the semantic predictive of the class. \n2. The definition of the \"minimal semantics\" $z_i$ is vague. The meaning of $Z_i$ is also unclear in Eq. (3).\n3. Could you provide a concrete implementation of L-Reg? I wonder which layer(s) is(are) chosen for computing Eq. (3) in practice? Are the results sensitive to the selection of layers?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a sample-based regularization technique, L-Reg, which goes beyond techniques like parameter-based L2 regularization by being more interpretable and demonstrating better generalization ability. The work formalizes the notion of semantic support to force the model to learn minimal sufficient statistics, quantitatively and qualitatively showing how that it leads to better generalization across multiple settings - multi-domain generalization, generalized category discovery and a new setting that is a combination of the two which they introduce.\n\nSTRENGTHS:\n+ A theoretically grounded paper with comprehensive experiments and results. \n+ The paper is well written in general. \n+ I especially liked how the algebraic logic formalism was neatly tied into this space. The idea of using semantic supports, although simple, is motivated and formalized well. I also liked how the negation of the semantic support set was used to formulate the optimization problem. \n+ The derivation of conditions required to hold under various settings is well done and makes the derivation of the regularization easy to follow. The proposed mDG + GCD setting is interesting.\n\nWEAKNESSES:\n* Although the method introduced is technically sound, with the baselines being quite comprehensive, the improvement in results seems minor. This suggests that accuracy may not be the right metric to compare against here. Considering the objectives proposed, shouldn’t other metrics beyond accuracy be considered?\n* The paper states that “the semantics generated by the encoder and classifier can be combined to form atomic formulas”. I expected to see some of the actual learnt atomic formulae in the results - which I did not.\n* From the formulation of the L_reg loss, it appears that any concept-based model that is sparse may be similar to the proposed formulation. How is the proposed method different from such methods -- formally and empirically?\n\nQUESTIONS:\n* Since the paper mentions “constructing atomic formulae”, would it be possible to actually extract and see how these look from the model? This might significantly strengthen the paper.\n* How does this method compare, formally and empirically, with sparse concept-based models? Wouldn’t their loss be very similar to the proposed L_reg loss?\n\nLIMITATIONS:\nYes, limitations have been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper proposes a novel logical regularization termed L-Reg for visual classification. L-Reg encourages models to focus on the salient semantics and thereby emerges interpretability. The theoretical analysis provides clear connections between logical reasoning and L-Reg. Extensive experiments demonstrate that L-Reg also benefits the generalization of models to unseen domains and categories.

STRENGTHS:
1.	Studies on loss regularization have positive influences on various fields.
2.	The paper is well-presented, and L-Reg is clearly presented with rigorous theoretical analysis.
3.	The qualitative benefits of L-Reg are validated through experiments, and the generalization brought by L-Reg has been demonstrated sufficiently under three settings.

WEAKNESSES:
1.	As stated in the introduction section, interpretability is a longstanding focus among studies in regularization terms. The authors claim that L2 regularization might lead to ambiguous interpretability, which also serves as an important motivation and contribution for L-Reg. However, the interpretability of L-Reg has not been adequately discussed except for the introduction section. Further analysis from either qualitative or quantitative perspectives can greatly strengthen the presentation.
2.	The analysis of Figure 5 should be detailed, especially the examples of unknown classes in Row 3.

QUESTIONS:
See weakness.

LIMITATIONS:
The author have discussed both the limitations and potential societal impact. The limitations are left for future research.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly focuses on two problems: 1) How does logical reasoning relate to visual tasks such as image classification? 2) How can we derive a logical reasoning-based regularization term to benefit generalization?. Then, this paper proposes a method called Logical Reasoning Regularization based on the analysis of the two problems. Theoretical analysis and experimental results demonstrate that L-Reg enhances generalization across several scenarios.\n\nSTRENGTHS:\n1. The main contributions of this article are: 1) Building the relationship between logical reasoning and visual tasks such as image classification; 2) Rethinking the classification task from the logical reasoning perspective and proposing Logical Reasoning Regularization. Overall, the contributions are meaningful, and the paper is interesting.\n2. The paper is easy to read.\n\nWEAKNESSES:\n1. As can be seen from Table 1 and Table 2, the proposed regular term has a weak improvement on the existing method.\n2. The analysis of this paper is incomplete and lacks theoretical analysis of sufficient conditions or sufficient and necessary conditions for meeting Atomic Formulas.\n3. Overclaim. This paper claims that the proposed L-Reg can reduce the complexity reduction. However, the proposed L-Reg is regarded as a regular term and is directly added in the learning objective. This does not reduce the computational complexity of the training period, but the extra regular term increases the computational overhead of the whole training process.\n\nQUESTIONS:\n1. line 123, f_s should be f.\n2. In Figure 1, it can be seen from the visualization results of the first line that the learned features are basically concentrated on the background, and it is felt that the obtained model is overfitting or underfitting. I would like to know the specific parameter Settings, experimental codes, random seeds, etc., of the experiment.\n3. In Figure 3, what does the abscissa represent? What is the baseline being compared? To make the results more convincing, an additional baseline and dataset need to be added.\n4. In Figure 4, the coordinate proportions of the left subgraph and the right subgraph are inconsistent, so it is unfair to make a direct comparison. Secondly, the horizontal and vertical coordinates need to be explained what they represent. Finally, the left and right subgraphs are almost identical, It cannot be proved that +L-Reg can achieve real elimination of certain extracted semantics characterized by dominant frequencies across all samples.\n5. In lines 173-186, Why the constraint in eq. (6) is removed in eq. (8), please give a detailed derivation process.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}