Training Data Protection with Compositional Diffusion Models
Authors: name: Aditya Golatkar, name: Alessandro Achille, name: Ashwin Swaminathan, name: Stefano Soatto

We introduce Compartmentalized Diffusion Models (CDM), a method to train
different diffusion models (or prompts) on distinct data sources and
arbitrarily compose them at inference time. The individual models can be
trained in isolation, at different times, and on different distributions and
domains and can be later composed to achieve performance comparable to a
paragon model trained on all data simultaneously. Furthermore, each model only
contains information about the subset of the data it was exposed to during
training, enabling several forms of training data protection. In particular,
CDMs are the first method to enable both selective forgetting and continual
learning for large-scale diffusion models, as well as allowing serving
customized models based on the user's access rights. CDMs also allow
determining the importance of a subset of the data in generating particular
samples.