---BEGIN ESSAY---

Why AI isn't "influenced by" an image in the way you think...

I recently had a discussion with someone where they were convinced that if a model didn't know how to represent a "centaur" and then you trained it on pictures of centaurs, then you could identify that every time the model generated a picture of centaurs, the primary influence was those (say, for sake of argument, 100) pictures of centaurs.

This sounds naively rational and seems like it should be true if you don't understand how neural networks work. So I think it's important to clarify why this is absolutely false and how much more complex AI models are than people seem to think they are.

For simplicity's sake, I'm going to talk about traditional "classifier" type models. The things I'm going to say extend to modern, transformer-based, cross-attention, generative models, but that's a whole lot of complexity to get through for the average person.
What Are Layers?

[For a really good, but fairly approachable summary of this, see the video I recently posted here, "What Do Neural Networks Really Learn?"]

When a model is trained, it naturally creates layers of abstraction. If you want a model to be able to identify different kinds of subjects in images, for example, then you will need a first layer of neurons that identify simplistic shapes; then a layer that identifies more complex structures made of those shapes; and then a layer that identifies even more abstract features using those structures.

This is how classifier models (and indeed, just about every modern AI model) work. Each layer of neurons identifies and responds to more and more abstract elements of the input, so that by the end, the result is "centaur" or "horse" or "man".

But here's the tricky question:
So There's a "Centaur Neuron" Trained by Centaur Images?

It's natural to then assume that if you take an existing model and train it on new features, such as centaurs, that it will just grow a new "centaur neuron" at the final layer and every time a centaur picture comes in, that neuron will be the one to give the strongest response.

That's... sort of true. In a traditional classifier (there are other ways to do this) you would add a new output neuron for "centaur" but that neuron only knows how to assemble the knowledge of previous input layers. It can't do anything with a raw image of a centaur. Most of the learning that is required to identify a centaur was already done ahead of time, and all the network did was wire up the new output along with some trivial jiggling of the rest of the network.
So How Do Models Work and How Can I Trace the Influence of an Input?

In reality, models build structures that react to complex blends of features. The same nodes in the layer that flags centaur features might also flag parrots and slugs. You cannot reliably trace these complexities through even a moderately sized network because it's just too complex with too many edge cases. Did the images of centaurs have the strongest influence on centaur identification? Probably not. At a guess, I'd say that images of horses probably had a stronger influence. The key is just to re-use those structures and identify where relatively simple differences between horses and centaurs occur. So most of the heavy lifting was already done, and you just needed to know how to bring the identification of humans and horses together.

(in generative AI, the same principle holds; the ability to render horse features won't be re-learned for centaurs and most of the neural network will be relatively unaffected by chimeral creatures, since the required features were already available, though what the network learns about centaurs might not all be obvious at first... for example, centaurs are probably most often going to be associated with fantasy art, a genre with fairly specific influences, and the AI doesn't know that you mean "the body shape" not "the genre of art")
Conclusion and TL;DR

AI models are insanely complex layers of understanding of their inputs, built up over a very long period of training that accretes structures to other structures in whatever configuration best suits the need. To make simple pronouncements of, "this image had the greatest influence on this result," is far too simplistic, and would not be practical in any real-world situation.

---END ESSAY---

Critique the above essay.