What's in a Latent? Leveraging Diffusion Latent Space for Domain
  Generalization
Authors: name: Xavier Thomas; name: Deepti Ghadiyaram
Published: 2025-03-09 17:29:01

Abstract: Domain Generalization aims to develop models that can generalize to novel and
unseen data distributions. In this work, we study how model architectures and
pre-training objectives impact feature richness and propose a method to
effectively leverage them for domain generalization. Specifically, given a
pre-trained feature space, we first discover latent domain structures, referred
to as pseudo-domains, that capture domain-specific variations in an
unsupervised manner. Next, we augment existing classifiers with these
complementary pseudo-domain representations making them more amenable to
diverse unseen test domains. We analyze how different pre-training feature
spaces differ in the domain-specific variances they capture. Our empirical
studies reveal that features from diffusion models excel at separating domains
in the absence of explicit domain labels and capture nuanced domain-specific
information. On 5 datasets, we show that our very simple framework improves
generalization to unseen domains by a maximum test accuracy improvement of over
4% compared to the standard baseline Empirical Risk Minimization (ERM).
Crucially, our method outperforms most algorithms that access domain labels
during training.