Self-Organising Textures

### Contents

[Patterns, textures and physical processes](#patterns-textures-and-physical-processes)
* [From Turing, to Cellular Automata, to Neural Networks](#from-turing-to-cellular-automata-to-neural-networks)
* [NCA as pattern generators](#nca-as-pattern-generators)
* [Related work](#related-work)

[Feature Visualization](#feature-visualization)
* [NCA with Inception](#nca-with-inception)

[Other interesting findings](#other-interesting-findings)
* [Robustness](#robustness)
* [Hidden States](#hidden-states)

[Conclusion](#conclusion)

![](images/multiple-pages.svg)

 This article is part of the
 [Differentiable Self-organizing Systems Thread](/2020/selforg/),
 an experimental format collecting invited short articles delving into
 differentiable self-organizing systems, interspersed with critical
 commentary from several experts in adjacent fields.

[Self-classifying MNIST Digits](/2020/selforg/mnist/)
[Adversarial Reprogramming of Neural Cellular Automata](/selforg/2021/adversarial/)

Neural Cellular Automata (NCA We use NCA to refer to both *Neural Cellular Automata* and *Neural Cellular Automaton*.) are capable of learning a diverse set of behaviours: from generating stable, regenerating, static images , to segmenting images , to learning to “self-classify” shapes . The inductive bias imposed by using cellular automata is powerful. A system of individual agents running the same learned local rule can solve surprisingly complex tasks. Moreover, individual agents, or cells, can learn to coordinate their behavior even when separated by large distances. By construction, they solve these tasks in a massively parallel and inherently degenerate Degenerate in this case refers to the [biological concept of degeneracy](https://en.wikipedia.org/wiki/Degeneracy_(biology)). way. Each cell must be able to take on the role of any other cell - as a result they tend to generalize well to unseen situations.

In this work, we apply NCA to the task of texture synthesis. This task involves reproducing the general appearance of a texture template, as opposed to making pixel-perfect copies. We are going to focus on texture losses that allow for a degree of ambiguity. After training NCA models to reproduce textures, we subsequently investigate their learned behaviors and observe a few surprising effects. Starting from these investigations, we make the case that the cells learn distributed, local, algorithms.

To do this, we apply an old trick: we employ neural cellular automata as a differentiable image parameterization .

Patterns, textures and physical processes
-----------------------------------------

![](images/zebra.jpg)
A pair of Zebra. Zebra are said to have unique stripes.

Zebra stripes are an iconic texture. Ask almost anyone to identify zebra stripes in a set of images, and they will have no trouble doing so. Ask them to describe what zebra stripes look like, and they will gladly tell you that they are parallel stripes of slightly varying width, alternating in black and white. And yet, they may also tell you that no two zebra have the same set of stripes Perhaps an apocryphal claim, but at the very lowest level every zebra will be unique. Ourp point is - “zebra stripes” as a concept in human understanding refers to the general structure of a black and white striped pattern and not to a specific mapping from location to colour.. This is because evolution has programmed the cells responsible for creating the zebra pattern to generate a pattern of a certain quality, with certain characteristics, as opposed to programming them with the blueprints for an exact bitmap of the edges and locations of stripes to be moulded to the surface of the zebra’s body.

Put another way, patterns and textures are ill-defined concepts. The Cambridge English Dictionary defines a pattern as “any regularly repeated arrangement, especially a design made from repeated lines, shapes, or colours on a surface”. This definition falls apart rather quickly when looking at patterns and textures that impart a feeling or quality, rather than a specific repeating property. A coloured fuzzy rug, for instance, can be considered a pattern or a texture, but is composed of strands pointing in random directions with small random variations in size and color, and there is no discernable regularity to the pattern. Penrose tilings do not repeat (they are not translationally invariant), but show them to anyone and they’ll describe them as a pattern or a texture. Most patterns in nature are outputs of locally interacting processes that may or may not be stochastic in nature, but are often based on fairly simple rules. There is a large body of work on models which give rise to such patterns in nature; most of it is inspired by Turing’s seminal paper on morphogenesis.

Such patterns are very common in developmental biology . In addition to coat colors and skin pigmentation, invariant large-scale patterns, arising in spite of stochastic low-level dynamics, are a key feature of peripheral nerve networks, vascular networks, somites (blocks of tissue demarcated in embryogenesis that give rise to many organs), and segments of anatomical and genetic-level features, including whole body plans (e.g., snakes and centipedes) and appendages (such as demarcation of digit fields within the vertebrate limb). These kinds of patterns are generated by reaction-diffusion processes, bioelectric signaling, planar polarity, and other cell-to-cell communication mechanisms. Patterns in biology are not only structural, but also physiological, as in the waves of electrical activity in the brain and the dynamics of gene regulatory networks. These gene regulatory networks, for example, can support computation sufficiently sophisticated as to be subject to Liar paradoxes See [liar paradox](https://en.wikipedia.org/wiki/Liar_paradox). In principle, gene regulatory networks can express paradoxical behaviour, such as that expression of factor A represses the expression of factor A.  One result of such a paradox can be that a certain factor will oscillate with time. . Studying the emergence and control of such patterns can help us to understand not only their evolutionary origins, but also how they are recognized (either in the visual system of a second observer or in adjacent cells during regeneration) and how they can be modulated for the purposes of regenerative medicine.

As a result, when having any model learn to produce textures or patterns, we want it to learn a generative process for the pattern. We can think of such a process as a means of sampling from the distribution governing this pattern. The first hurdle is to choose an appropriate loss function, or qualitative measure of the pattern. To do so, we employ ideas from Gatys et. al . NCA become the parametrization for an image which we “stylize” in the style of the target pattern. In this case, instead of restyling an existing image, we begin with a fully unconstrained setting: the output of an untrained, randomly initialized, NCA. The NCA serve as the “renderer” or “generator”, and a pre-trained differentiable model serves as a distinguisher of the patterns, providing the gradient necessary for the renderer to learn to produce a pattern of a certain style.

### From Turing, to Cellular Automata, to Neural Networks

NCA are well suited for generating textures. To understand why, we’ll demonstrate parallels between texture generation in nature and NCA. Given these parallels, we argue that NCA are a good model class for texture generation.

#### PDEs

In “The Chemical Basis of Morphogenesis” , Alan Turing suggested that simple physical processes of reaction and diffusion, modelled by partial differential equations, lie behind pattern formation in nature, such as the aforementioned zebra stripes. Extensive work has since been done to identify PDEs modeling reaction-diffusion and evaluating their behaviour. One of the more celebrated examples is the Gray-Scott model of reaction diffusion (,). This process has a veritable zoo of interesting behaviour, explorable by simply tuning the two parameters. We strongly encourage readers to visit this [interactive atlas](http://mrob.com/pub/comp/xmorphia/) of the different regions of the Gray-Scott reaction diffusion model to get a sense for the extreme variety of behaviour hidden behind two simple knobs. The more adventurous can even [play with a simulation locally](https://groups.csail.mit.edu/mac/projects/amorphous/jsim/sim/GrayScott.html) or [in the browser](https://mrob.com/pub/comp/xmorphia/ogl/index.html).

To tackle the problem of reproducing our textures, we propose a more general version of the above systems, described by a simple Partial Differential Equation (PDE) over the state space of an image.

∂s∂t=f(s,∇xs,∇x2s)\frac{\partial \mathbf{s} }{\partial t } = f(\textbf{s}, \nabla\_\mathbf{x} \textbf{s}, \nabla\_\mathbf{x}^{2}\textbf{s})∂t∂s=f(s,∇xs,∇x2s)

Here, fff is a function that depends on the gradient (∇xs\nabla\_\mathbf{x} \textbf{s}∇xs) and Laplacian (∇x2s\nabla\_\mathbf{x}^{2}\textbf{s}∇x2s) of the state space and determines the time evolution of this state space. sss represents a k dimensional vector, whose first three components correspond to the visible RGB color channels.

Intuitively, we have defined a system where every point of the image changes with time, in a way that depends on how the image currently changes across space, with respect to its immediate neighbourhood. Readers may start to recognize the resemblance between this and another system based on immediately local interactions.

#### To CAs

Differential equations governing natural phenomena are usually evaluated using numerical differential equation solvers. Indeed, this is sometimes the **only** way to solve them, as many PDEs and ODEs of interest do not have closed form solutions. This is even the case for some deceptively simple ones, such as the [three-body problem](https://en.wikipedia.org/wiki/Three-body_problem). Numerically solving PDEs and ODEs is a vast and well-established field. One of the biggest hammers in the metaphorical toolkit for numerically evaluating differential equations is discretization: the process of converting the variables of the system from continuous space to a discrete space, where numerical integration is tractable. When using some ODEs to model a change in a phenomena over time, for example, it makes sense to advance through time in discrete steps, possibly of variable size.

We now show that numerically integrating the aforementioned PDE is equivalent to reframing the problem as a Neural Cellular Automata, with fff assuming the role of the NCA rule.

The logical approach to discretizing the space the PDE operates on is to discretize the continuous 2D image space into a 2D raster grid. Boundary conditions are of concern but we can address them by moving to a toroidal world where each dimension wraps around on itself.

Similarly to space, we choose to treat time in a discretized fashion and evaluate our NCA at fixed-sized time steps. This is equivalent to explicit Euler integration. However, here we make an important deviation from traditional PDE numerical integration methods for two reasons. First, if all cells are updated synchronously, initial conditions s0s\_0s0 must vary from cell-to-cell in order to break the symmetry. Second, the physical implementation of the synchronous model would require the existence of a global clock, shared by all cells. One way to work around the former is by initializing the grid with random noise, but in the spirit of self organisation we instead choose to decouple the cell updates by asynchronously evaluating the CA. We sample a subset of all cells at each time-step to update. This introduces both asynchronicity in time (cells will sometimes operate on information from their neighbours that is several timesteps old), and asymmetry in space, solving both aforementioned issues.

Our next step towards representing a PDE with cellular automata is to discretize the gradient and Laplacian operators. For this we use the [sobel operator](https://en.wikipedia.org/wiki/Sobel_operator) and the [9-point variant](https://en.wikipedia.org/wiki/Discrete_Laplace_operator) of the discrete Laplace operator, as below.

[−101−202−101][−1−2−1000121][1212−122121]SobelxSobelyLaplacian \begin{array}{ c c c }
\begin{bmatrix}
-1 & 0 & 1\\-2 & 0 & 2 \\-1 & 0 & 1
\end{bmatrix}
&
\begin{bmatrix}
-1 & -2 & -1\\ 0 & 0 & 0 \\1 & 2 & 1
\end{bmatrix}
&
\begin{bmatrix}
1 & 2 & 1\\2 & -12 & 2 \\1 & 2 & 1
\end{bmatrix}
\\
Sobel\_x & Sobel\_y & Laplacian
\end{array}⎣⎡−1−2−1000121⎦⎤Sobelx⎣⎡−101−202−101⎦⎤Sobely⎣⎡1212−122121⎦⎤Laplacian

With all the pieces in place, we now have a space-discretized version of our PDE that looks very much like a Cellular Automata: the time evolution of each discrete point in the raster grid depends only on its immediate neighbours. These discrete operators allow us to formalize our PDE as a CA. To double check that this is true, simply observe that as our grid becomes very fine, and the asynchronous updates approach uniformity, the dynamics of these discrete operators will reproduce the continuous dynamics of the original PDE as we defined it.

#### To Neural Networks

The final step in implementing the above general PDE for texture generation is to translate it to the language of deep learning. Fortunately, all the operations involved in iteratively evaluating the generalized PDE exist as common operations in most deep learning frameworks. We provide both a Tensorflow and a minimal PyTorch implementation for reference, and refer readers to these for details on our implementation.

### NCA as pattern generators

#### Model:

![](images/texture_model.svg)
Texture NCA model.

We build on the Growing CA NCA model , complete with built-in quantization of weights, stochastic updates, and the batch pool mechanism to approximate long-term training. For further details on the model and motivation, we refer readers to this work.

#### Loss function:

![](images/texture_training.svg)
Texture NCA model.

We use a well known deep convolutional network for image recognition, VGG (Visual Geometry Group Net ) as our differentiable discriminator of textures, for the same reasons outlined in Differentiable Parametrizations . We start with a template image, x⃗\vec{x}x⃗, which we feed into VGG. Then we collect statistics from certain layers (block[1...51…51...5]\_conv1) in the form of the raw activation values of the neurons in these layers. Finally, we run our NCA forward for between 32 and 64 iterations, feeding the resulting RGB image into VGG. Our loss is the L2L\_2L2 distance between the gram matrix For a brief definition of gram matrices, see [here](https://www.tensorflow.org/tutorials/generative/style_transfer#calculate_style). of activations of these neurons with the NCA as input and their activations with the template image as input. We keep the weights of VGG frozen and use ADAM  to update the weights of the NCA.

#### Dataset:

The template images for this dataset are from the Oxford Describable Textures Dataset . The aim of this dataset is to provide a benchmark for measuring the ability of vision models to recognize and categorize textures and describe textures using words. The textures were collected to match 47 “attributes” such as “bumpy” or “polka-dotted”. These 47 attributes were in turn distilled from a set of common words used to describe textures identified by Bhusan, Rao and Lohse .

#### Results:

After a few iterations of training, we see the NCA converge to a solution that at first glance looks similar to the input template, but not pixel-wise identical. The very first thing to notice is that the solution learned by the NCA is **not** time-invariant if we continue to iterate the CA. In other words it is constantly changing!

This is not completely unexpected. In *Differentiable Parametrizations*, the authors noted that the images produced when backpropagating into image space would end up different each time the algorithm was run due to the stochastic nature of the parametrizations. To work around this, they introduced some tricks to maintain **alignment** between different visualizations. In our model, we find that we attain such alignment along the temporal dimension without optimizing for it; a welcome surprise. We believe the reason is threefold. First, reaching and maintaining a static state in an NCA appears to be non-trivial in comparison to a dynamic one, so much so that in Growing CA a pool of NCA states at various iteration times had to be maintained and sampled as starting states to simulate loss being applied after a time period longer than the NCAs iteration period, to achieve a static stability. We employ the same sampling mechanism here to prevent the pattern from decaying, but in this case the loss doesn’t enforce a static fixed target; rather it guides the NCA towards any one of a number of states that minimizes the style loss. Second, we apply our loss after a random number of iterations of the NCA. This means that, at any given time step, the pattern must be in a state that minimizes the loss. Third, the stochastic updates, local communication, and quantization all limit and regularize the magnitude of updates at each iteration. This encourages changes to be small between one iteration and the next. We hypothesize that these properties combined encourage the NCA to find a solution where each iteration is **aligned** with the previous iteration. We perceive this alignment through time as motion, and as we iterate the NCA we observe it traversing a manifold of locally aligned solutions.

We now **posit** *that finding temporally aligned solutions is equivalent to finding an algorithm, or process, that generates the template pattern*, based on the aforementioned findings and qualitative observation of the NCA. We proceed to demonstrate some exciting behaviours of NCA trained on different template images.

An NCA trained to create a pattern in the style of **chequered\_0121.jpg**.

Here, we see that the NCA is trained using a template image of a simple black and white grid.

We notice that:

* Initially, a non-aligned grid of black and white quadrilaterals is formed.
* As time progresses, the quadrilaterals seemingly grow or shrink in both x⃗\vec{x}x⃗ and y⃗\vec{y}y⃗ to more closely approximate squares. Quadrilaterals of both colours either emerge or disappear. Both of these behaviours seem to be an attempt to find local consistency.
* After a longer time, the grid tends to achieve perfect consistency.

Such behaviour is not entirely unlike what one would expect in a hand-engineered algorithm to produce a consistent grid with local communication. For instance, one potential hand-engineered approach would be to have cells first try and achieve local consistency, by choosing the most common colour from the cells surrounding them, then attempting to form a diamond of correct size by measuring distance to the four edges of this patch of consistent colour, and moving this boundary if it were incorrect. Distance could be measured by using a hidden channel to encode a gradient in each direction of interest, with each cell decreasing the magnitude of this channel as compared to its neighbour in that direction. A cell could then localize itself within a diamond by measuring the value of two such gradient channels. The appearance of such an algorithm would bear resemblance to the above - with patches of cells becoming either black, or white, diamonds then resizing themselves to achieve consistency.

An NCA trained to create a pattern in the style of **bubbly\_0101.jpg**.

In this video, the NCA has learned to reproduce a texture based on a template of clear bubbles on a blue background. One of the most interesting behaviours we observe is that the density of the bubbles remains fairly constant. If we re-initialize the grid states, or interactively destroy states, we see a multitude of bubbles re-forming. However, as soon as two bubbles get too close to each other, one of them spontaneously collapses and disappears, ensuring a constant density of bubbles throughout the entire image. We regard these bubbles as ”[solitons](#an-aside-solitons-and-lenia)″ in the solution space of our NCA. This is a concept we will discuss and investigate at length below.

If we speed the animation up, we see that different bubbles move at different speeds, yet they never collide or touch each other. Bubbles also maintain their structure by self-correcting; a damaged bubble can re-grow.

This behaviour is remarkable because it arises spontaneously, without any external or auxiliary losses. All of these properties are learned from a combination of the template image, the information stored in the layers of VGG, and the inductive bias of the NCA. The NCA learned a rule that effectively approximates many of the properties of the bubbles in the original image. Moreover, it has learned a process that generates this pattern in a way that is robust to damage and looks realistic to humans.

An NCA trained to create a pattern in the style of **interlaced\_0172.jpg**.

Here we see one of our favourite patterns: a simple geometric “weave”. Again, we notice the NCA seems to have learned an algorithm for producing this pattern. Each “thread” alternately joins or detaches from other threads in order to produce the final pattern. This is strikingly similar to what one would attempt to implement, were one asked to programmatically generate the above pattern. One would try to design some sort of stochastic algorithm for weaving individual threads together with other nearby threads.

An NCA trained to create a pattern in the style of **banded\_0037.jpg**.

Here, misaligned stripe fragments travel up or down the stripe until either they merge to form a single straight stripe or a stripe shrinks and disappears. Were this to be implemented algorithmically with local communication, it is not infeasible that a similar algorithm for finding consistency among the stripes would be used.

### Related work

This foray into pattern generation is by no means the first. There has been extensive work predating deep-learning, in particular suggesting deep connections between spatial patterning of anatomical structure and temporal patterning of cognitive and computational processes (e.g., reviewed in ). Hans Spemann, one of the heroes of classical developmental biology, said “Again and again terms have been used which point not to physical but to psychical analogies. It was meant to be more than a poetical metaphor. It was meant to my conviction that the suitable reaction of a germ fragment, endowed with diverse potencies, in an embryonic ‘field’… is not a common chemical reaction, like all vital processes, are comparable, to nothing we know in such degree as to vital processes of which we have the most intimate knowledge.” . More recently, Grossberg quantitatively laid out important similarities between developmental patterning and computational neuroscience . As briefly touched upon, the inspiration for much of the work came from Turing’s work on pattern generation through local interaction, and later papers based on this principle. However, we also wish to acknowledge some works that we feel have a particular kinship with ours.

#### Patch sampling

Early work in pattern generation focused on texture sampling. Patches were often sampled from the original image and reconstructed or rejoined in different ways to obtain an approximation of the texture. This method has also seen recent success with the work of Gumin .

#### Deep learning

Gatys et. al’s work , referenced throughout, has been seminal with regards to the idea that statistics of certain layers in a pre-trained network can capture textures or styles in an image. There has been extensive work building on this idea, including playing with other parametrisations for image generation  and optimizing the generation process .

Other work has focused on using a convolutional generator combined with path sampling and trained using an adversarial loss to produce textures of similar quality .

#### Interactive Evolution of Camouflage

Perhaps the most unconventional approach, with which we find kinship, is laid out in *Interactive Evolution of Camouflage* . Craig Reynolds uses a texture description language, consisting of generators and operators, to parametrize a texture patch, which is presented to human viewers who have to decide which patches are the worst at “camouflaging” themselves against a chosen background texture. The population is updated in an evolutionary fashion to maximize “camouflage”, resulting in a texture exhibiting the most camouflage (to human eyes) after a number of iterations. We see strong parallels with our work - instead of a texture generation language, we have an NCA parametrize the texture, and instead of human reviewers we use VGG as an evaluator of the quality of a generated pattern. We believe a fundamental difference lies in the solution space of an NCA. A texture generation language comes with a number of inductive biases and learns a deterministic mapping from coordinates to colours. Our method appears to learn more general algorithms and behaviours giving rise to the target pattern.

Two other noteworthy examples of similar work are Portilla et. al’s work with the wavelet transform , and work by Chen et al with reaction diffusion .

Feature visualization
---------------------

![](images/butterfly_eye.jpg)
A butterfly with an “eye-spot” on the wings.

We have now explored some of the fascinating behaviours learned by the NCA when presented with a template image. What if we want to see them learn even more “unconstrained” behaviour?

Some butterflies have remarkably lifelike eyes on their wings. It’s unlikely the butterflies are even aware of this incredible artwork on their own bodies. Evolution placed these there to trigger a response of fear in potential predators or to deflect attacks from them . It is likely that neither the predator nor the butterfly has a concept for what an eye is or what an eye does, or even less so any [theory of mind](https://en.wikipedia.org/wiki/Theory_of_mind) regarding the consciousness of the other, but evolution has identified a region of morphospace for this organism that exploits pattern-identifying features of predators to trick them into fearing a harmless bug instead of consuming it.

Even more remarkable is the fact that the individual cells composing the butterfly’s wings can self assemble into coherent, beautiful, shapes far larger than an individual cell - indeed a cell is on the order of 1−5m1^{-5}m1−5m  while the features on the wings will grow to as large as 1−3m1^{-3}m1−3m . The coordination required to produce these features implies self-organization over hundreds or thousands of cells to generate a coherent image of an eye that evolved simply to act as a visual stimuli for an entirely different species, because of the local nature of cell-to-cell communication. Of course, this pales in comparison to the morphogenesis that occurs in animal and plant bodies, where structures consisting of millions of cells will specialize and coordinate to generate the target morphology.

A common approach to investigating neural networks is to look at what inhibits or excites individual neurons in a network . Just as neuroscientists and biologists have often treated cells and cell structures and neurons as black-box models to be investigated, measured and reverse-engineered, there is a large contemporary body of work on doing the same with neural networks. For instance the work by Boettiger  .

We can explore this idea with minimal effort by taking our pattern-generating NCA and exploring what happens if we task it to enter a state that excites a given neuron in Inception. One of the common resulting NCAs we notice is eye and eye-related shapes - such as the video below - likely as a result of having to detect various animals in ImageNet. In the same way that cells form eye patterns on the wings of butterflies to excite neurons in the brains of predators, our NCA’s population of cells has learned to collaborate to produce a pattern that excites certain neurons in an external neural network.

An NCA trained to excite **mixed4a\_472** in Inception.

### NCA with Inception

#### Model:

We use a model identical to the one used for exploring pattern generation, but with a different discriminator network: Imagenet-trained Inception v1 network .

#### Loss function:

Our loss maximizes the activations of chosen neurons, when evaluated on the output of the NCA. We add an auxiliary loss to encourage the outputs of the NCA to be ∈[0,1]\in [0,1]∈[0,1], as this is not inherently built into the model. We keep the weights of the Inception frozen and use ADAM  to update the weights of the NCA.

#### Dataset:

There is no explicit dataset for this task. Inception is trained on ImageNet. The layers and neurons we chose to excite are chosen qualitatively using OpenAI Microscope.

#### Results:

Similar to the pattern generation experiment, we see quick convergence and a tendency to find temporally dynamic solutions. In other words, resulting NCAs do not stay still. We also observe that the majority of the NCAs learn to produce solitons of various kinds. We discuss a few below, but encourage readers to explore them in the demo.

An NCA trained to excite **mixed4c\_439** in Inception.

Solitons in the form of regular circle-like shapes with internal structure are quite commonly observed in the inception renderings. Two solitons approaching each other too closely may cause one or both of them to decay. We also observe that solitons can divide into two new solitons.

An NCA trained to excite **mixed3b\_454** in Inception.

In textures that are composed of threads or lines, or in certain excitations of Inception neurons where the resulting NCA has a “thread-like” quality, the threads grow in their respective directions and will join other threads, or grow around them, as required. This behaviour is similar to the regular lines observed in the striped patterns during pattern generation.

Other interesting findings
--------------------------

### Robustness

#### Switching manifolds

We encode local information flow within the NCA using the same fixed Laplacian and gradient filters. As luck would have it, these can be defined for most underlying manifolds, giving us a way of placing our cells on various surfaces and in various configurations without having to modify the learned model. Suppose we want our cells to live in a hexagonal world. We can redefine our kernels as follows:

![](images/hex_kernels.svg)
Hexagonal grid convolutional filters.

Our model, trained in a purely square environment, works out of the box on a hexagonal grid! Play with the corresponding setting in the demo to experiment with this. Zooming in allows observation of the individual hexagonal or square cells. As can be seen in the demo, the cells have no problem adjusting to a hexagonal world and producing identical patterns after a brief period of re-alignment.

![](images/coral_square.png)
![](images/coral_hex.png)
The same texture evaluated on a square and hexagonal grid, respectively.

#### Rotation

![](images/mond_rot0.png)
![](images/mond_rot1.png)
![](images/mond_rot2.png)
![](images/mond_rot3.png)
Mondrian pattern where the cells are rotated in various directions. Note that the NCA is not re-trained - it gen-
eralises to this new rotated paradigm without issue.

In theory, the cells can be evaluated on any manifold where one can define approximations to the Sobel kernel and the Laplacian kernel. We demonstrate this in our demo by providing an aforementioned “hexagonal” world for the cells to live in. Instead of having eight equally-spaced neighbours, each cell now has six equally-spaced neighbours. We further demonstrate this versatility by rotating the Sobel and Laplacian kernels. Each cell receives an innate global orientation based on these kernels, because they are defined with respect to the coordinate system of the state. Redefining the Sobel and Laplacian kernel with a rotated coordinate system is straightforward and can even be done on a per-cell level. Such versatility is exciting because it mirrors the extreme robustness found in biological cells in nature. Cells in most tissues will generally continue to operate whatever their location, direction, or exact placement relative to their neighbours. We believe this versatility in our model could even extend to a setting where the cells are placed on a manifold at random, rather than on an ordered grid.

#### Time-synchronization

Two NCAs running next to each other, at different speeds, with some stochasticity in speed. They can communicate through their shared edge; the vertical boundary between them in the center of the state space.

Stochastic updates teach the cells to be robust to asynchronous updates. We investigate this property by taking it to an extreme and asking *how* *do the cells react if two manifolds are allowed to communicate but one runs the NCA at a different speed than the other*? The result is surprisingly stable; the CA is still able to construct and maintain a consistent texture across the combined manifold. The time discrepancy between the two CAs sharing the state is far larger than anything the NCA experiences during training, showing remarkable robustness of the learned behaviour. Parallels can be drawn to organic matter self repairing, for instance a fingernail can regrow in adulthood despite the underlying finger already having fully developed; the two do not need to be sync. This result also hints at the possibility of designing distributed systems without having to engineer for a global clock, synchronization of compute units or even homogenous compute capacity.

An NCA is evaluated for a number of steps. The surrounding border of cells are then also turned into NCA cells. The cells have no difficulty communicating with the “finished” pattern and achieving consistency.

An even more drastic example of this robustness to time asynchronicity can be seen above. Here, an NCA is iterated until it achieves perfect consistency in a pattern. Then, the state space is expanded, introducing a border of new cells around the existing state. This border quickly interfaces with the existing cells and settles in a consistent pattern, with almost no perturbation to the already-converged inner state.

#### Failure cases

The failure modes of a complex system can teach us a great deal about its internal structure and process. Our model has many quirks and sometimes these prevent it from learning certain patterns. Below are some examples.

![](images/fail_mondrian.jpeg)
![](images/fail_sprinkle.jpeg)
![](images/fail_chequerboard.jpeg)
Three failure cases of the NCA. Bottom row shows target texture samples, top row are corresponding NCA outputs. Failure modes include incorrect colours, chequerboard artefacts, and incoherent image structure.

Some patterns are reproduced somewhat accurately in terms of structure, but not in colour, while some are the opposite. Others fail completely. It is difficult to determine whether these failure cases have their roots in the parametrization (the NCA), or in the hard-to-interpret gradient signals from VGG, or Inception. Existing work with style transfer suggests that using a loss on Gram matrices in VGG can introduce instabilities , that are similar to the ones we see here. We hypothesize that this effect explains the failures in reproducing colors. The structural failures, meanwhile, may be caused by the NCA parameterization, which makes it difficult for cells to establish long-distance communication with one another.

### Hidden states

When biological cells communicate with each other, they do so through a multitude of available communication channels. Cells can emit or absorb different ions and proteins, sense physical motion or “stiffness” of other cells, and even emit different chemical signals to diffuse over the local substrate .

There are various ways to visualize communication channels in real cells. One of them is to add to cells a potential-activated dye. Doing so gives a clear picture of the voltage potential the cell is under with respect to the surrounding substrate. This technique provides useful insight into the communication patterns within groups of cells and helps scientists visualize both local and global communication over a variety of time-scales.

As luck would have it, we can do something similar with our Neural Cellular Automata. Our NCA model contains 12 channels. The first three are visible RGB channels and the rest we treat as latent channels which are visible to adjacent cells during update steps, but excluded from loss functions. Below we map the first three principle components of the hidden channels to the R,G, and B channels respectively. Hidden channels can be considered “floating,” to abuse a term from circuit theory. In other words, they are not pulled to any specific final state or intermediate state by the loss. Instead, they converge to some form of a dynamical system which assists the cell in fulfilling its objective with respect to its visible channels. There is no pre-defined assignment of different roles or meaning to different hidden channels, and there is almost certainly redundancy and correlation between different hidden channels. Such correlation may not be visible when we visualize the first three principal components in isolation. But this concern aside, the visualization yields some interesting insights anyways.

**Left:** RGB channels of NCA. **Right:** Intensities of top three principal components of hidden states.

 An NCA trained to excite **mixed4b\_70** in Inception. Notice the hidden states appear to encode information about structure. “Threads” along the major diagonal (NW - SE) appear primarily green, while those running along the anti-diagonal appear blue, indicating that these have differing internal states, despite being effectively indistinguishable in RGB space.

In the principal components of this coral-like texture, we see a pattern which is similar to the visible channels. However, the “threads” pointing in each diagonal direction have different colours - one diagonal is green and the other is a pale blue. This suggests that one of the things encoded into the hidden states is the direction of a “thread”, likely to allow cells that are inside one of these threads to keep track of which direction the thread is growing, or moving, in.

**Left:** RGB channels of NCA. **Right:** Intensities of top three principal components of hidden states.

 An NCA trained to produce a texture based on DTD image **cheqeuered\_0121**. Notice the structure of squares - with a gradient occurring inside the structure of each square, evidencing that structure is being encoded in hidden state.

The chequerboard pattern likewise lends itself to some qualitative analysis and hints at a fairly simple mechanism for maintaining the shape of squares. Each square has a clear gradient in PCA space across the diagonal, and the values this gradient traverses differ for the white and black squares. We find it likely the gradient is used to provide a local coordinate system for creating and sizing the squares.

**Left:** RGB channels of NCA. **Right:** Intensities of top three principal components of hidden states.

 An NCA trained to excite **mixed4c\_208** in Inception. The visible body of the eye is clearly demarcated in the hidden states. There is also a “halo” which appears to modulate growth of any solitons immediately next to each other. This halo is barely visible in the RGB channels.

We find surprising insight in NCA trained on Inception as well. In this case, the structure of the eye is clearly encoded in the hidden state with the body composed primarily of one combination of principal components, and an halo, seemingly to prevent collisions of the eye solitons, composed of another set of principal components.

Analysis of these hidden states is something of a dark art; it is not always possible to draw rigorous conclusions about what is happening. We welcome future work in this direction, as we believe qualitative analysis of these behaviours will be useful for understanding more complex behaviours of CAs. We also hypothesize that it may be possible to modify or alter hidden states in order to affect the morphology and behaviour of NCA.

Conclusion
----------

In this work, we selected texture templates and individual neurons as targets and then optimized NCA populations so as to produce similar excitations in a pre-trained neural network. This procedure yielded NCAs that could render nuanced and hypnotic textures. During our analysis, we found that these NCAs have interesting and unexpected properties. Many of the solutions for generating certain patterns in an image appear similar to the underlying model or physical behaviour producing the pattern. For example, our learned NCAs seem to have a bias for treating objects in the pattern as individual objects and letting them move freely across space. While this effect was present in many of our models, it was particularly strong in the bubble and eye models. The NCA is forced to find algorithms that can produce such a pattern with purely local interaction. This constraint seems to produce models that favor high-level consistency and robustness.

![](images/multiple-pages.svg)

 This article is part of the
 [Differentiable Self-organizing Systems Thread](/2020/selforg/),
 an experimental format collecting invited short articles delving into
 differentiable self-organizing systems, interspersed with critical
 commentary from several experts in adjacent fields.

[Self-classifying MNIST Digits](/2020/selforg/mnist/)
[Adversarial Reprogramming of Neural Cellular Automata](/selforg/2021/adversarial/)

Eyvind Niklasson
Alexander Mordvintsev
Michael Levin

Neural Cellular Automata learn to generate textures, exhibiting surprising properties.

Growing Neural Cellular Automata

Image segmentation via Cellular Automata

Self-classifying MNIST Digits

Differentiable Image Parameterizations

The chemical basis of morphogenesis

Turing patterns in development: what about the horse part?

A unified design space of synthetic stripe-forming networks

On the Formation of Digits and Joints during Limb Development

Modeling digits. Digit patterning is controlled by a Bmp-Sox9-Wnt Turing network modulated by morphogen gradients

Pattern formation mechanisms of self-organizing reaction-diffusion systems

Bioelectric gene and reaction networks: computational modelling of genetic, biochemical and bioelectrical dynamics in pattern regulation

Turing-like patterns can arise from purely bioelectric mechanisms

Dissipative structures in biological systems: bistability, oscillations, spatial patterns and waves

Gene networks and liar paradoxes

Texture Synthesis Using Convolutional Neural Networks

The chemical basis of morphogenesis. 1953

Pattern formation by interacting chemical fronts

Complex patterns in a simple system

Very Deep Convolutional Networks for Large-Scale Image Recognition

Adam: A Method for Stochastic Optimization

Describing Textures in the Wild

The texture lexicon: Understanding the categorization of visual texture terms and their relationship to texture images

Re-membering the body: applications of computational neuroscience to the top-down control of regeneration of limbs and other complex organs

Embryonic Development and Induction

Communication, Memory, and Development

WaveFunctionCollapse

Texture Networks: Feed-forward Synthesis of Textures and Stylized Images

TextureGAN: Controlling deep image synthesis with texture patches

Interactive evolution of camouflage

A parametric texture model based on joint statistics of complex wavelet coefficients

Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration

The evolutionary significance of butterfly eyespots

Live Cell Imaging of Butterfly Pupal and Larval Wings In Vivo

Focusing on butterfly eyespot focus: uncoupling of white spots from eyespot bodies in nymphalid butterflies

OpenAI Microscope

The neural origins of shell structure and pattern in aquatic mollusks

Emergent complexity in simple neural systems

Going deeper with convolutions

Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses

Stem cell migration and mechanotransduction on linear stiffness gradient hydrogels


Visualizing Weights

![](images/multiple-pages.svg)

 This article is part of the [Circuits thread](/2020/circuits/), an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks.

[Curve Circuits](/2020/circuits/curve-circuits/)
[Branch Specialization](/2020/circuits/branch-specialization/)

Introduction
------------

The problem of understanding a neural network is a little bit like reverse engineering a large compiled binary of a computer program. In this analogy, the weights of the neural network are the compiled assembly instructions. At the end of the day, the weights are the fundamental thing you want to understand: how does this sequence of convolutions and matrix multiplications give rise to model behavior?

Trying to understand artificial neural networks also has a lot in common with neuroscience, which tries to understand biological neural networks. As you may know, one major endeavor in modern neuroscience is mapping the [connectomes](https://en.wikipedia.org/wiki/Connectome) of biological neural networks: which neurons connect to which. These connections, however, will only tell neuroscientists which weights are non-zero. Getting the weights – knowing whether a connection excites or inhibits, and by how much – would be a significant further step. One imagines neuroscientists might give a great deal to have the access to weights that those of us studying artificial neural networks get for free.

And so, it’s rather surprising how little attention we actually give to looking at the weights of neural networks. There are a few exceptions to this, of course. It’s quite common for researchers to show pictures of the first layer weights in vision models (these are directly connected to RGB channels, so they’re easy to understand as images). In some work, especially historically, we see researchers reason about the weights of toy neural networks by hand. And we quite often see researchers discuss aggregate statistics of weights. But actually looking at the weights of a neural network other than the first layer is quite uncommon – to the best of our knowledge, mapping weights between hidden layers to meaningful algorithms is novel to the circuits project.

What’s the difference between visualizing activations, weights, and attributions?
---------------------------------------------------------------------------------

In this article, we’re focusing on visualizing weights. But people often visualize activations, attributions, gradients, and much more. How should we think about the meaning of visualizing these different objects?

* **Activations:** We generally think of these as being “what” the network saw. If understanding a neural network is like reverse compiling a computer program, the neurons are the variables, and the activations are the values of those variables.
* **Weights:** We generally think of these as being “how” the neural network computes one layer from the previous one. In the reverse engineering analogy, these are compiled assembly instructions.
* **Attributions:** Attributions try to tell us the extent to which one neuron influenced a later neuron.We often think of this as “why” the neuron fired. We need to be careful with attributions, because they’re a human-defined object on top of a neural network rather than a fundamental object. They aren’t always well defined, and people mean different things by them. (They are very well defined if you are only operating across adjacent layers!)

Why it’s non-trivial to study weights in hidden layers
------------------------------------------------------

It seems to us that there are three main barriers to making sense of the weights in neural networks, which may have contributed to researchers tending to not directly inspect them:

* **Lack of Contextualization:** Researchers often visualize weights in the first layer, because they are linked to RGB values that we understand. That connection makes weights in the first layer meaningful. But weights between hidden layers are meaningless by default: knowing nothing about either the source or the destination, how can we make sense of them?
* **Indirect Interaction:** Sometimes, the meaningful weight interactions are between neurons which aren’t literally adjacent in a neural network. For example, in a residual network, the output of one neuron can pass through the additive residual stream and linearly interact with a neuron much later in the network. In other cases, neurons may interact through intermediate neurons without significant nonlinear interactions. How can we efficiently reason about these interactions?
* **Dimensionality and Scale:** Neural networks have lots of neurons. Those neurons connect to lots of other neurons. There’s a lot of data to display! How can we reduce it to a human-scale amount of information?

Many of the methods we’ll use to address these problems were previously explored in [Building Blocks](https://distill.pub/2018/building-blocks/) in the context of understanding activation vectors. The goal of this article is to show how similar ideas can be applied to weights instead of activations. Of course, we’ve already implicitly used these methods in various circuit articles, but in those articles the methods have been of secondary interest to the results. It seems useful to give some dedicated discussion to the methods.

Aside: One Simple Trick
-----------------------

Interpretability methods often fail to take off because they’re hard to use. So before diving into sophisticated approaches, we wanted to offer a simple, easy to apply method.

In a convolutional network, the input weights for a given neuron have shape `[width, height, input_channels]`. Unless this is the first convolutional layer, this probably can’t be easily visualized because `input_channels` is large. (If this is the first convolutional layer, visualize it as is!) However, one can use dimensionality reduction to collapse `input_channels` down to 3 dimensions. We find one-sided NMF especially effective for this.

![](images/screenshot_1.png)

[1](#figure-1):
 NMF of input weights in InceptionV1 `mixed4d_5x5`, for a selection of ten neurons. The red, green, and blue channels on each grid indicate the weights for each of the 3 NMF factors.

This visualization doesn’t tell you very much about what your weights are doing in the context of the larger model, but it does show you that they are learning nice spatial structures. This can be an easy sanity check that your neurons are learning, and a first step towards understanding your neuron’s behavior. We’ll also see later that this general approach of factoring weights can be extended into a powerful tool for studying neurons.

Despite this lack of contextualization, one-sided NMF can be a great technique for investigating multiple channels at a glance. One thing you may quickly discover using this method is that, in models with global average pooling at the end of their convolutional layers, the last few layers will have all their weights be horizontal bands.

![](images/screenshot_2.png)

[2](#figure-2):
 Horizontally-banded weights in InceptionV1 `mixed5b_5x5`, for a selection of eight neurons. As in Figure 1, the red, green, and blue channels on each grid indicate the weights for each of the 3 NMF factors.

We call this phenomenon [*weight banding*](/2020/circuits/weight-banding/). One-sided NMF allows for quickly testing and validating hypotheses about phenomena such as weight banding.

Contextualizing Weights with Feature Visualization
--------------------------------------------------

Of course, looking at weights in a vacuum isn’t very interesting. In order to really understand what’s going on, we need to *contextualize* weights in the broader context of the network. The challenge of contextualization is a recurring challenge in understanding neural networks: we can easily observe every activation, every weight, and every gradient; the challenge lies in determining what those values represent.

Recall that the weights between two convolutional layers are a four dimensional array of the shape:

`[relative x position, relative y position,
 input channels, output channels]`

If we fix the input channel and the output channel, we get a 2D array we can present with traditional data visualization. Let’s assume we know which neuron we’re interested in understanding, so we have the output channel. We can pick the input channels with high magnitude weights to our output channel.

But what does the input represent? What about the output?

The key trick is that techniques like feature visualization (or deeper investigations of neurons) can help us understand what the input and output neurons represent, contextualizing the graph. Feature visualizations are especially attractive because they’re automatic, and produce a single image which is often very informative about the neuron. As a result, we often represent neurons as feature visualizations in weight diagrams.

[3](#figure-3): Contextualizing weights.

This approach is the weight analogue of using feature visualizations to contextualize activation vectors in [Building Blocks](https://distill.pub/2018/building-blocks/) (see the section titled “Making Sense of Hidden Layers”).

We can liken this to how, when reverse-engineering a normal compiled computer program, one would need to start assigning variable names to the values stored in registers to keep track of them. Feature visualizations are essentially automatic variable names for neurons, which are roughly analogous to those registers or variables.

### Small Multiples

Of course, neurons have multiple inputs, and it can be helpful to show the weights to several inputs at a time as a [small multiple](https://en.wikipedia.org/wiki/Small_multiple):

[4](#figure-4): Small multiple weights for [`mixed3b` 342](https://microscope.openai.com/models/inceptionv1/mixed3b_0/342).

And if we have two families of related neurons interacting, it can sometimes even be helpful to show the weights between all of them as a grid of small multiples:

[5](#figure-5): Small multiple weights for a variety of [curve detectors](https://distill.pub/2020/circuits/curve-detectors/).

Advanced Approaches to Contextualization with Feature Visualization
-------------------------------------------------------------------

Although we most often use feature visualization to visualize neurons, we can visualize any direction (linear combination of neurons). This opens up a very wide space of possibilities for visualizing weights, of which we’ll explore a couple particularly useful ones.

### Visualizing Spatial Position Weights

Recall that the weights for a single neuron have shape `[width, height, input_channels]`. In the previous section we split up `input_channels` and visualized each `[width, height]` matrix. But an alternative approach is to think of there as being a vector over input neurons at each spatial position, and to apply feature visualization to each of those vectors. You can think of this as telling us what the weights in that position are collectively looking for.

![](images/screenshot_6.png)

[6](#figure-6). **Left:** Feature visualization of a car neuron. **Right:** Feature visualizations of the vector over input neurons at each spatial position of the car neuron’s weights. As we see, the car neuron broadly responds to window features above wheel features.

This visualization is the weight analogue of the [“Activation Grid” visualization](https://distill.pub/2018/building-blocks/#ActivationVecVis) from Building Blocks. It can be a nice, high density way to get an overview of what the weights for one neuron are doing. However, it will be unable to capture cases where one position responds to multiple very different things, as in a multi-faceted or polysemantic neuron.

### Visualizing Weight Factors

Feature visualization can also be applied to factorizations of the weights, which we briefly discussed earlier. This is the weight analogue to the “Neuron Groups” visualization from Building Blocks.

This can be especially helpful when you have a group of neurons like [high-low frequency detectors](/2020/circuits/frequency-edges/) or black and white vs color detectors that look are all mostly looking for a small number of factors. For example, a large number of high-low frequency detectors can be significantly understood as combining just two factors – a high frequency factor and a low-frequency factor – in different patterns.

 .upstream-nmf {
 display: grid;
 grid-row-gap: .5rem;
 margin-bottom: 2rem;
 }

 .upstream-nmf .row {
 display: grid;
 grid-template-columns: min-content 1fr 6fr;
 grid-column-gap: 1rem;
 grid-row-gap: .5rem;

 }

 .units,
 .weights {
 display: grid;
 grid-template-columns: repeat(6, 1fr);
 grid-gap: 0.5rem;
 grid-column-start: 3;
 }

 img.fv {
 max-width: 100%;
 border-radius: 8px;
 }

 div.units img.full {
 margin-left: 1px;
 margin-bottom: 0px;
 }

 img.full {
 width: unset;
 object-fit: none;
 object-position: center;
 image-rendering: optimizeQuality;
 }

 img.weight {
 width: 100%;
 image-rendering: pixelated;
 align-self: center;
 border: 1px solid #ccc;
 }

 .annotated-image {
 display: grid;
 grid-auto-flow: column;
 align-items: center;
 }

 .annotated-image span {
 writing-mode: vertical-lr;
 }

 .layer-label {
 grid-row-start: span 2;
 text-align: end;
 }

 .layer-label label {
 display: inline-block;
 writing-mode: vertical-lr;
 }

 .layer-label.hidden {
 border-color: transparent;
 }

 .layer-label.nonhidden {
 margin-left: 1rem;
 }

 .layer-label.hidden label {
 visibility: hidden;
 }

mixed3a

[![](diagrams/figure-7-feature-vis/neuron-136.png "Unit 136")](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_136.html)
[![](diagrams/figure-7-feature-vis/neuron-108.png "Unit 108")](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_108.html)
[![](diagrams/figure-7-feature-vis/neuron-132.png "Unit 132")](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_132.html)
[![](diagrams/figure-7-feature-vis/neuron-88.png "Unit 88")](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_88.html)
[![](diagrams/figure-7-feature-vis/neuron-110.png "Unit 110")](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_110.html)
[![](diagrams/figure-7-feature-vis/neuron-180.png "Unit 180")](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_180.html)

HF-factor
![](diagrams/figure-8-upstream-nmf/conv2d2-hi.png)

![](diagrams/figure-8-upstream-nmf/neuron=136-layer=maxpool1-factor=1.png)
![](diagrams/figure-8-upstream-nmf/neuron=108-layer=maxpool1-factor=1.png)
![](diagrams/figure-8-upstream-nmf/neuron=132-layer=maxpool1-factor=1.png)
![](diagrams/figure-8-upstream-nmf/neuron=88-layer=maxpool1-factor=1.png)
![](diagrams/figure-8-upstream-nmf/neuron=110-layer=maxpool1-factor=1.png)
![](diagrams/figure-8-upstream-nmf/neuron=180-layer=maxpool1-factor=1.png)

LF-factor
![](diagrams/figure-8-upstream-nmf/conv2d2-lo.png)

![](diagrams/figure-8-upstream-nmf/neuron=136-layer=maxpool1-factor=0.png)
![](diagrams/figure-8-upstream-nmf/neuron=108-layer=maxpool1-factor=0.png)
![](diagrams/figure-8-upstream-nmf/neuron=132-layer=maxpool1-factor=0.png)
![](diagrams/figure-8-upstream-nmf/neuron=88-layer=maxpool1-factor=0.png)
![](diagrams/figure-8-upstream-nmf/neuron=110-layer=maxpool1-factor=0.png)
![](diagrams/figure-8-upstream-nmf/neuron=180-layer=maxpool1-factor=0.png)

[7](#figure-7):

 NMF factorization on the weights (excitatory and inhibitory) connecting six high-low frequency detectors in InceptionV1 to the
 layer `conv2d2`.

These factors can then be decomposed into individual neurons for more detailed understanding.

 .upstream-neurons {
 display: grid;
 grid-gap: 1em;
 margin-bottom: 1em;
 }

 h5 {
 margin-bottom: 0px;
 }

 .upstream-neurons .row {
 display: grid;
 grid-template-columns: 1fr min-content 1fr min-content 1fr min-content 1fr min-content 1fr min-content 1fr min-content min-content;
 grid-column-gap: .25em;
 column-gap: .25em;
 align-items: center;
 }

 .units,
 .weights {
 display: grid;
 grid-template-columns: repeat(6, 1fr);
 grid-gap: 0.5rem;
 grid-column-start: 3;
 }

 img.fv {
 display: block;
 max-width: 100%;
 border-radius: 8px;
 }

 img.full {
 width: unset;
 object-fit: none;
 object-position: center;
 image-rendering: optimizeQuality;
 }

 img.weight {
 width: 100%;
 image-rendering: pixelated;
 align-self: center;
 border: 1px solid #ccc;
 }

 .layer-label {
 grid-row-start: span 2;
 }

 .layer-label label {
 display: inline-block;
 /\* transform: rotate(-90deg); \*/
 }

 .annotation {
 font-size: 1.5em;
 font-weight: 200;
 color: #666;
 margin-bottom: 0.2em;
 }

 .equal-sign {
 padding: 0 0.25em;
 }

 .ellipsis {
 padding: 0 0.25em;
 /\* vertically align the ellipsis \*/
 position: relative;
 bottom: 0.5ex;
 }

 .unit {
 display: block;
 min-width: 50px;
 }

 .factor {
 box-shadow: 0 0 8px #888;
 }

 .unit .bar {
 display: block;
 margin-top: 0.5em;
 background-color: #CCC;
 height: 4px;
 }

 .row h4 {
 border-bottom: 1px solid #ccc;
 }

![](diagrams/figure-8-upstream-nmf/conv2d2-hi.png)
=
[![](diagrams/figure-8-upstream-neurons/layer_name=conv2d2-component=0-channel_index=119.png)](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_119.html)
+
[![](diagrams/figure-8-upstream-neurons/layer_name=conv2d2-component=0-channel_index=102.png)](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_102.html)
+
[![](diagrams/figure-8-upstream-neurons/layer_name=conv2d2-component=0-channel_index=123.png)](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_123.html)
+
[![](diagrams/figure-8-upstream-neurons/layer_name=conv2d2-component=0-channel_index=90.png)](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_90.html)
+
[![](diagrams/figure-8-upstream-neurons/layer_name=conv2d2-component=0-channel_index=173.png)](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_173.html)
+
…
HF-factor

 × 0.93

 × 0.73

 × 0.66

 × 0.59

 × 0.55

![](diagrams/figure-8-upstream-nmf/conv2d2-lo.png)
=
[![](diagrams/figure-8-upstream-neurons/layer_name=conv2d2-component=1-channel_index=89.png)](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_89.html)
+
[![](diagrams/figure-8-upstream-neurons/layer_name=conv2d2-component=1-channel_index=163.png)](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_163.html)
+
[![](diagrams/figure-8-upstream-neurons/layer_name=conv2d2-component=1-channel_index=98.png)](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_98.html)
+
[![](diagrams/figure-8-upstream-neurons/layer_name=conv2d2-component=1-channel_index=188.png)](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_188.html)
+
[![](diagrams/figure-8-upstream-neurons/layer_name=conv2d2-component=1-channel_index=158.png)](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_158.html)
+
…
LF-factor

 × 0.44

 × 0.41

 × 0.38

 × 0.36

 × 0.34

[8](#figure-8): Neurons (by their feature visualizations) that contribute to the two NMF factors from [Figure 7](#figure-7), plus the weighted amount they contribute to
 each factor.

Dealing with Indirect Interactions
----------------------------------

As we mentioned earlier, sometimes the meaningful weight interactions are between neurons which aren’t literally adjacent in a neural network, or where the weights aren’t directly represented in a single weight tensor. A few examples:

* In a residual network, the output of one neuron can pass through the additive residual stream and linearly interact with a neuron much later in the network.
* In a separable convolution, weights are stored as two or more factors, and need to be expanded to link neurons.
* In a bottleneck architecture, neurons in the bottleneck may primarily be a low-rank projection of neurons from the previous layer.
* The behavior of an intermediate layer simply doesn’t introduce much non-linear behavior, leaving two neurons in non-adjacent layers with a significant linear interaction.

As a result, we often work with “expanded weights” – that is, the result of multiplying adjacent weight matrices, potentially ignoring non-linearities. We generally implement expanded weights by taking gradients through our model, ignoring or replacing all non-linear operations with the closest linear one.

These expanded weights have the following properties:

* If two layers interact **linearly**, the expanded weights will give the true linear map, even if the model doesn’t explicitly represent the weights in a single weight matrix.
* If two layers interact **non-linearly**, the expanded weights can be seen as the expected value of the gradient up to a constant factor, under the assumption that all neurons have an equal (and independent) probability of firing.

They also have one additional benefit, which is more of an implementation detail: because they’re implemented in terms of gradients, you don’t need to know how the weights are represented. For example, in TensorFlow, you don’t need to know which variable object represents the weights. This can be a significant convenience when you’re working with unfamiliar models!

### Benefits of Expanded Weights

Multiplying out the weights like this can sometimes help us see a simpler underlying structure. For example, [`mixed3b` 208](https://microscope.openai.com/models/inceptionv1/mixed3b_0/208) is a black and white center detector. It’s built by combining a bunch of black and white vs color detectors together.

[9](#figure-9). `mixed3b` 208 along with five neurons from `mixed3a` that contribute the [strongest weights](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_208.html) to it.

Expanding out the weights allows us to see an important aggregate effect of these connections: together, they look for the absence of color in the center one layer further back.

[10](#figure-10). Top eighteen [expanded weights](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_208.html) from `conv2d2` to [`mixed3b` 208](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_208.html), organized in two rows according to weight factorization.

A particularly important use of this method – which we’ve been implicitly using in earlier examples – is to jump over “bottleneck layers.” Bottleneck layers are layers of the network which squeeze the number of channels down to a much smaller number, typically in a branch, making large spatial convolutions cheaper. The [bottleneck layers](https://microscope.openai.com/models/inceptionv1/mixed3a_5x5_bottleneck_0?models.op.feature_vis.type=channel&models.op.technique=feature_vis) of InceptionV1 are one example. Since so much information is compressed, these layers are often polysemantic, and it can often be more helpful to jump over them and understand the connection to the wider layer before them.

### Cases where expanded weights are misleading

Expanded weights can, of course, be misleading when non-linear structure is important. A good example of this is [boundary detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_boundary). Recall that boundary detectors usually detect both low-to-high and high-to-low frequency transitions:

[11](#figure-11). Boundary detectors such as [`mixed3b` 345](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_345.html) detect both low-to-high and high-to-low frequency transitions.

Since high-low frequency detectors are [usually](/2020/circuits/frequency-edges/) excited by high-frequency patterns on one side and inhibited on the other (and vice versa for low frequency), detecting both directions means that the expanded weights cancel out! As a result, expanded weights appear to show that boundary detectors are neither excited or inhibited by high frequency detectors two layers back, when in fact they are *both* excited and also inhibited by high frequency, depending on the context, and it’s just that those two different cases cancel out.

[12](#figure-12).

 Neurons two layers back (such as [`conv2d2` 89](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_89.html)) may have a strong influence on the high-low frequency detectors that contribute to `mixed3b` 345 (top), but that influence washes out when we look at the expanded weights (bottom) directly between `conv2d2` 89 and `mixed3b` 345.

More sophisticated techniques for describing multi-layer interactions can help us understand cases like this. For example, one can determine what the “best case” excitation interaction between two neurons is (that is, the maximum achievable gradient between them). Or you can look at the gradient for a particular example. Or you can factor the gradient over many examples to determine major possible cases. These are all useful techniques, but we’ll leave them for a future article to discuss.

### Qualitative properties

One qualitative property of expanding weights across many layers deserves mention before we end our discussion of them. Expanded weights often get this kind of “electron orbital”-like smooth spatial structures:

[13](#figure-13). Smooth spatial structure of some expanded weights from [`mixed3b` 268](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_268.html) to `conv2d1`.

 Although the exact structures present may vary from neuron to neuron, this example is not cherry-picked: this smoothness is typical of most multiple-layer expanded weights.

It’s not clear how to interpret this, but it’s suggestive of rich spatial structure on the scale of multiple layers.

Dimensionality and Scale
------------------------

So far, we’ve addressed the challenges of contextualization and indirection interactions. But we’ve only given a bit of attention to our third challenge of dimensionality and scale. Neural networks contain many neurons and each one connects to many others, creating a huge amount of weights. How do we pick which connections between neurons to look at?

For the purposes of this article, we’ll put the question of which neurons we want to study outside of our scope, and only discuss the problem of picking which connections to study. (We may be trying to comprehensively study a model, in which case we want to study all neurons. But we might also, for example, be trying to study neurons we’ve determined related to some narrower aspect of model behavior.)

Generally, we chose to look at the largest weights, as we did at the beginning of the section on contextualization. Unfortunately, there tends to be a long tail of small weights, and at some point it generally gets impractical to look at these. How much of the story is really hiding in these small weights? We don’t know, but polysemantic neurons suggest there could be a very important and subtle story hiding here! There’s some hope that sparse neural networks might make this much better, by getting rid of small weights, but whether such conclusions can be drawn about non-sparse networks is presently speculative.

An alternative strategy that we’ve brushed on a few times is to reduce your weights into a few components and then study those factors (for example, with NMF). Often, a very small number of components can explain much of the variance. In fact, sometimes a small number of factors can explain the weights of an entire set of neurons! Prominent examples of this are high-low frequency detectors (as we saw earlier) and black and white vs color detectors.

However, this approach also has downsides. Firstly, these components can be harder to understand and even polysemantic. For example, if you apply the basic version of this method to a boundary detector, one component will contain both high-to-low and low-to-high frequency detectors which will make it hard to analyze. Secondly, your factors no longer align with activation functions, which makes analysis much messier. Finally, because you will be reasoning about every neuron in a different basis, it is difficult to build a bigger picture view of the model unless you convert your components back to neurons.

![](images/multiple-pages.svg)

 This article is part of the [Circuits thread](/2020/circuits/), an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks.

[Curve Circuits](/2020/circuits/curve-circuits/)
[Branch Specialization](/2020/circuits/branch-specialization/)

Chelsea Voss
Nick Cammarata
Gabriel Goh
Michael Petrov
Ludwig Schubert
Swee Kiat Lim
Chris Olah

We present techniques for visualizing, contextualizing, and understanding neural network weights.

Understanding neural networks through deep visualization

Visualizing and understanding convolutional networks

The Building Blocks of Interpretability

Zoom In: An Introduction to Circuits

An Overview of Early Vision in InceptionV1

Curve Detectors

Feature Visualization

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Visualizing and understanding recurrent networks

Visualizing higher-layer features of a deep network