DreamDiffusion: Generating High-Quality Images from Brain EEG Signals
Authors: name: Yunpeng Bai; name: Xintao Wang; name: Yanpei Cao; name: Yixiao Ge; name: Chun Yuan; name: Ying Shan
Published: 2023-06-29 13:33:02

Abstract: This paper introduces DreamDiffusion, a novel method for generating
high-quality images directly from brain electroencephalogram (EEG) signals,
without the need to translate thoughts into text. DreamDiffusion leverages
pre-trained text-to-image models and employs temporal masked signal modeling to
pre-train the EEG encoder for effective and robust EEG representations.
Additionally, the method further leverages the CLIP image encoder to provide
extra supervision to better align EEG, text, and image embeddings with limited
EEG-image pairs. Overall, the proposed method overcomes the challenges of using
EEG signals for image generation, such as noise, limited information, and
individual differences, and achieves promising results. Quantitative and
qualitative results demonstrate the effectiveness of the proposed method as a
significant step towards portable and low-cost ``thoughts-to-image'', with
potential applications in neuroscience and computer vision.