Summarize this:
A fundamental goal of computer vision is to construct artificial systems that see and recognize the world as human visual systems do. Recent developments in the measurement of population brain activity, combined with advances in the implementation and design of deep neural network models, have allowed direct comparisons between latent representations in biological brains and architectural characteristics of artificial networks, providing important insights into how these systems operate [3, 8–10, 13, 18, 19, 21, 42, 43, 54, 55]. These efforts have included the reconstruction of visual experiences (perception or imagery) from brain activity, and the examination of potential correspondences between the computational processes associated with biological and artificial systems [2, 5, 7, 24, 25, 27, 36, 44–46].

Reconstructing visual images from brain activity, such as that measured by functional Magnetic Resonance Imaging (fMRI), is an intriguing but challenging problem, because the underlying representations in the brain are largely unknown, and the sample size typically associated with brain data is relatively small [17, 26, 30, 32]. In recent years, researchers have started addressing this task using deep-learning models and algorithms, including generative adversarial networks (GANs) and self-supervised learning [2, 5, 7, 24, 25, 27, 36, 44–46]. Additionally, more recent studies have increased semantic fidelity by explicitly using the semantic content of images as auxiliary inputs for reconstruction [5, 25]. However, these studies require training new generative models with fMRI data from scratch, or fine-tuning toward the specific stimuli used in the fMRI experiment. These efforts have shown impressive but limited success in pixel-wise and semantic fidelity, partly because the number of samples in neuroscience is small, and partly because learning complex generative models poses numerous challenges.

Diffusion models (DMs) [11,47,48,53] are deep generative models that have been gaining attention in recent years. DMs have achieved state-of-the-art performance in several tasks involving conditional image generation [4,39,49], image super resolution [40], image colorization [38], and other related tasks [6, 16, 33, 41]. In addition, recently proposed latent diffusion models (LDMs) [37] have further reduced computational costs by utilizing the latent space generated by their autoencoding component, enabling more efficient computations in the training and inference phases. Another advantage of LDMs is their ability to generate high-resolution images with high semantic fidelity. However, because LDMs have been introduced only recently, we still lack a satisfactory understanding of their internal mechanisms. Specifically, we still need to discover how they represent latent signals within each layer of DMs, how the latent representation changes throughout the denoising process, and how adding noise affects conditional image generation.

Here, we attempt to tackle the above challenges by reconstructing visual images from fMRI signals using an LDM named Stable Diffusion. This architecture is trained on a large dataset and carries high text-to-image generative performance. We show that our simple framework can reconstruct high-resolution images with high semantic fidelity without any training or fine-tuning of complex deeplearning models. We also provide biological interpretations of each component of the LDM, including forward/reverse diffusion processes, U-Net, and latent representations with different noise levels.

Our contributions are as follows: (i) We demonstrate that our simple framework can reconstruct high-resolution (512 × 512) images from brain activity with high semantic fidelity, without the need for training or fine-tuning of complex deep generative models (Figure 1); (ii) We quantitatively interpret each component of an LDM from a neuroscience perspective, by mapping specific components to distinct brain regions; (iii) We present an objective interpretation of how the text-to-image conversion process implemented by an LDM incorporates the semantic information expressed by the conditional text, while at the same time maintaining the appearance of the original image.