Kati8/ICLR_2023_2025 huggingface dataset:
authors: Di Zhao
authors: Gillian Dobbie
authors: Jingfeng Zhang
authors: Hongsheng Hu
authors: Philippe Fournier-Viger
authors: Yun Sing Koh
reviews: 
Comments: The authors present an approach to improve the generalizability of deep learning models across multiple domains. It introduces the Balanced Online Knowledge Distillation (BOLD) framework, which leverages CLIP-based model as the teacher model and extract its domain-specific and domain-invariant knowledge through an online distillation strategy. Extensive experiments on multiple benchmarks indicate the effectiveness of the proposed method.
Strengths: 1. The writting and the structure of the paper is clear. The idea of online knowledge distillation is interesting and seems to be effective.
2. The authors provides theoretical analysis.
3. The experiments are thorough.
Weaknesses: 1. The proposed method is based on CLIP output adapters, which is an already well investigated topic in the previous works [1]. Even though the proposed online knowledge distillation ($L_{spc}$) seems to be novel, but it is only a tiny component of the proposed method. From this point of view, the contribution of this work to the community might be limited. The authors should further discuss this.
2. What is the intuition of using cross-entropy loss in the expert training? Why not just using the similarity-based metrics for text-image matching?
3. The implementation of the existing methods is unclear. For instance, for the baseline RISE, they claim to achieve around 90.2 with ResNet-50 backbone on PACS dataset. But in Table 1 their performance is only 86.6. The author should further clarify this. Besides, the authors mentioned that they used the setting "ViT-B/32 to ResNet50" for Table 1, while RISE used ResNet-50 as the backbone. The authors should describe more experimental details regarding this for a fair comparison.

[1] Gao, Peng, et al. "Clip-adapter: Better vision-language models with feature adapters." International Journal of Computer Vision 132.2 (2024): 581-595.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: Current knowledge distillation-based domain generalization approaches overlook the importance of domain-specific knowledge and rely on a two-stage training process, which limits the effectiveness of knowledge transfer. To overcome these limitations, this paper proposes the Balanced Online knowLedge Distillation (BOLD) framework for domain generalization, exploring the domain invariant for effective knowledge transfer while domain-specific knowledge is reserved. Experiments demonstrates its effectiveness.
Strengths: 1. The codes are provided, which makes it easy to reproduce performance.

2. The question raised is meaningful. How to retain domain-specific information is a point worth exploring in the field of knowledge transfer.

3. Theoretical proof is provided, and the effectiveness of the method is analyzed theoretically.
Weaknesses: 1. There is some deficiency in related works. The exploration of domain-specific has been reflected in some domain adaptation/domain generalization literatures in the past, and it needs to be reflected in related works. e.g., [1][2][3].

2. The variant of ablation study was a little too simple, and we expected to see the effect of domain-invariant and domain-specific knowledge separately and the corresponding analysis.

[1] Bui M H, Tran T, Tran A, et al. Exploiting domain-specific features to enhance domain generalization[J]. Advances in Neural Information Processing Systems, 2021, 34: 21189-21201.

[2] Seo S, Suh Y, Kim D, et al. Learning to optimize domain specific normalization for domain generalization[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16. Springer International Publishing, 2020: 68-83.

[3] Chang W G, You T, Seo S, et al. Domain-specific batch normalization for unsupervised domain adaptation[C]//Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. 2019: 7354-7362.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: The paper propose the Balanced Online knowLedge Distillation (BOLD) framework for domain generalization, which employs a multi-domain expert teacher model, with each expert specializing in specific source domains to preserve domain-specific knowledge. This is the first investigation into the effectiveness of online knowledge distillation for domain generalization. The study also demonstrates that distilling both domain-invariant and domain-specific knowledge, rather than only domain-invariant knowledge, enhances model generalizability. Extensive experiments across seven domain generalization benchmarks validate the effectiveness of the proposed BOLD framework compared to state-of-the-art methods.
Strengths: 1. The writing of this work is good, and the explanation of the method is clear and easy to understand.
2. The authors provided theoretical analysis to support the proposed method.
Weaknesses: 1. The novelty is limited. In fact, learning both domain-invariant and domain-specific features is a common approach in domain generalization, with a lot of theoretical and experimental research already existing on it (e.g. Bui, Manh-Ha, et al. "Exploiting domain-specific features to enhance domain generalization." Advances in Neural Information Processing Systems 34 (2021)). The main difference between the author's work and previous research lies in the integration of this idea with knowledge distillation and different loss function design.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 4
reviews: 
Comments: The authors present an approach called Balanced Online Knowledge Distillation (BOLD) for domain generalization. BOLD uses a multi-domain expert teacher model to retain domain-specific knowledge and employs an online distillation strategy, allowing the teacher and student models to learn simultaneously. This setup enhances knowledge transfer and improves the model’s ability to generalize across unseen domains. Extensive experiments on seven benchmarks demonstrate the effectiveness of BOLD over state-of-the-art methods.
Strengths: 1. The clarity of the paper is commendable, and the Balanced Online Knowledge Distillation (BOLD) framework demonstrates effectiveness across seven benchmarks.
 
2. The authors provide both theoretical and empirical evidence to support the effectiveness of this method.
Weaknesses: 1. The theoretical analysis in this paper lacks rigor in establishing a strict upper bound for convergence and relies heavily on assumptions without concrete mathematical proofs. Specifically:
- Absence of Formal Proof for Upper Bound: The derivations, such as in Equations (11) and (13), introduce error terms  ϵ and ϵo ​but lack rigorous proof that these terms converge in the desired manner, resulting in an incomplete justification for the generalization bound.
- Reliance on Assumptions: The analysis assumes that incorporating domain-specific knowledge and applying online distillation will reduce domain discrepancy and empirical risk, yet these effects are only qualitatively described without mathematical substantiation or a precise convergence rate.
2.  The Balanced Online Knowledge Distillation (BOLD) framework is underexplored, as it does not account for the imbalanced dataset distribution across different domains. This limitation raises concerns about the assumption that all experts are well-pretrained.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3