Internal representations in Latent Spaces
After the pre-learning phase, I would like to provide an image to the model and see its representation at each level of the latent space. Could someone advice me how I can do this? I have not found an easy way.
Initially I don't need to see the representations in the entire encoder-decoder; just its part used for Diffusion.
I am a researcher who works in bringing architectures and algorithms from Neuroscience into Deep Learning. Here, I want to see the Representations and Operations (R & O) in each of the latent spaces. The mammalian visual system has optimized its R&O over millions of years and I have an inkling that some of it can enhance the current version of Latent/Stable Diffusion models.
Unni
KP Unnikrishnan, PhD
eNeuroLearn, Ann Arbor, MI
This is a follow-up to my earlier post; it is a copy of my LinkedIn post today.
Neuroscience can help Stable Diffusion
This is a follow-up to my last post on #latentspace representations in #stablediffusion models. I now have a clearer picture on how #neuroscience can help.
Here are a few images from the most helpful #github notebook by johnowhitaker and banacl:
The LHS is the input and the RHS is the output of the Autoencoder. The box in the middle shows the #latentspace representations of this parrot. Notice how you can look at each of the images within the box and recognize that it is a parrot. That is because the latent spaces of this #stablediffusion model is in what neuroscientists call retino-topic representations.
In the mammalian visual system, retino-topic representations disappear after the first stage (LGN). All "mammalian latent spaces" are in feature-extracted representations. Since this system has evolved over hundreds of millions of years, it has been optimized for #perception and #cognition.
I would like to posit that the current #stablediffusion models can become a lot more efficient by incorporating some "Neurosciene Inside". I plan to pursue this. If anyone is interested, please drop me a note via #linkedin or to kunnik at gmail dot com.
Activate to view larger image
