A super-efficient neural network or hypernet
By: Andrey Belkin
Tags: AI, Brain, Neural Networks

What does it take to create artificial intelligence? Which way to go to this goal? Many will confidently answer that the scientific community is already moving at full speed along this path. That with the next neural network models with even more parameters, with even more powerful neuromorphic computers, and even larger datasets for all occasions, we will break into the era of thinking machines. In my opinion, it looks like a race for a carrot and this state of affairs will become more obvious to everyone when the number of tunable parameters in the models becomes more than connections in the human brain, but the expected effect will not be achieved.

In this article, I will outline the way out of the vicious race for carrots, and talk about my path and developments in my research.

The first thing that needs to be addressed on the way to artificial intelligence is the problem of low training efficiency of neural networks. It is foolish to build a grand high-rise building without a foundation, or on a very bad foundation. Many researchers continue to develop their concepts and models of intelligence on this foundation, instead of focusing on the obvious problems: biological neural networks learn many times more efficiently than artificial neural networks.

Yes, biological neural networks, one might say, are in a constant continuous flow of information, continuously receiving signals from their receptors. In a day, an infant can receive a gigantic amount of information from his or her organs of sight and hearing, equivalent to video and audio data for many, many terabytes. But if we analyze this information in detail, then there is not so much information that is really useful for training, so to speak, “marked” information. We can say that a person learns on a very poor quality dataset, most of the examples are irrelevant and not marked up, some of the examples are very noisy, but at the same time, he learns quite successfully and quickly.

Apparently, computer neural networks do not correspond to biological ones or we train them in a completely different way from the way it happens in the biological nervous system.

It is necessary to concentrate on developing super-efficient neural networks, rather than increasing the number of learning parameters.

Moreover, the main focus should be on the speed of network learning, while quality or accuracy is now secondary.

Only a super-efficient neural network (SNN) can be the foundation and foundation for creating artificial intelligence.

Having SNN technology, you can move in different ways, like creating cognitive architectures based on graphs, hypergraphs, etc. plus SNN, and moving in a biological way repeating the architecture of the biological brain. I am a supporter of the biological approach, although I do not consider this path to be the only one in achieving the goal.

Roadmap towards AI

First stage. Creation of a model of sensory analyzers, primarily visual and auditory. In general, sensory analyzers of the biological nervous system should become the prototype of super-efficient neural networks (SNN). Second stage. Gaining knowledge about the mechanisms of instant, temporary, and long-term memory in the concept of SNN. Biological sensory analyzers easily work with context, only understanding the concept of memory in biological neural networks will allow us to adopt the ability to work with context, which will greatly expand the capabilities of the SNN. Stage three. Creation of models of motor networks. Despite the homogeneity of the cerebral cortex, motor networks are in some ways more complex than sensory analyzers, although there are some deep similarities. Therefore, this stage will not belong. The motor networks will include a model of the cerebellum, as a structure that memorizes and reproduces short time intervals for precise movements. And also a kind of reward mechanism to ensure the learning of correct movements, similar to the dopamine reward mechanisms in the biological brain. Fourth stage. One of the important forms of motor actions is speech. Perhaps, for this reason, this and the previous stages will be implemented simultaneously. At this stage, the speech will not be fully conscious. As a person, stages of speech development must be passed, such as humming, the pronunciation of individual sounds, then words. The maximum level of speech proficiency at this stage is naming the presented objects, describing scenes, answering simple questions, reading, or declaring the text. At this stage, the foundation for the network of the speech circle will be laid (I wrote about the speech circle in detail in the article “What is the mind”). Fifth stage. Creation of a model of basic emotions and needs. This stage will not be a clear-cut stage; many of its concepts will manifest themselves at earlier stages. The main fundamental emotion that needs to be modeled is the emotion of novelty. It is enough to create an evaluative mechanism of information novelty at various levels of abstraction and to perform modulation of network parameters on its basis. Curiosity is essential for effective learning. You also need to provide the models with the senses they need to control their needs. For example, a low battery in a robot could create a certain feeling of hunger for the robot, and the robot would seek to satisfy it, that is, it would look for a charger or power source. Or creating some kind of feeling of pain or discomfort so that the robot avoids traumatic situations. Another useful emotion that can be easily modeled is the attachment of various kinds. Affection is based on an imprinting mechanism. This will be useful for all sorts of helper robots or pet robots that are required to follow a master's and have devotion. A person will be more willing to interact with intelligence similar to him, therefore, a wider range of emotional mechanisms may appear. For example, anxiety and fear, sympathy, joy, and disappointment. As for the introduction of Isaac Asimov’s laws of robotics to the level of basic emotions, they have a high abstract character, which greatly complicates their modeling. First, the model must be trained so that it is possible to identify the representation of the images “Human” and “harm”, similar to how one can find “grandmother’s neurons” in a person. Then you can associate the emotional mechanism with the desire to avoid combining these images. It is difficult to say how effective such a method will be. Sixth stage. We can say that this stage is as difficult as the first, if not more difficult. Conditionally, it can be called the stage of the prefrontal cortex. Higher cognitive activity is assigned to the prefrontal cortex; we can say that judgment and long-term goal-setting are located. The prefrontal cortex controls the work of the emotional centers and centers of need, it regulates how much they influence behavior. Working at this stage requires in-depth knowledge of human behavior. Only at this stage can one reason and talk about consciousness and conscious behavior. To this stage, I include not only the creation of the prefrontal cortex but also several important biological neural networks and mechanisms. For example, the “interpreter” network, which seeks to justify any of our actions. The presence of such networks is clearly evident in experiments involving people with split-brain hemispheres.

Or the network responsible for the illusion of qualia, creating a sense of centralized perception of the world. Also, the mechanism of perceiving the boundaries of one’s body and actions, a violation of this mechanism in people can lead to an obsessive desire to get rid of a part of their body or the inability to distinguish their actions from those of another person.

We know that disturbances in various mental mechanisms can lead to various diseases on the autism spectrum or various forms of schizophrenia. On the way to artificial intelligence, it is destined to encounter examples of these deviations in the artificial psyche, but it also has a useful grain. Work on artificial intelligence will allow us to better understand the nature of human mental illness.

Hypercolumn network, or hypernet

We are now at the first stage of this roadmap, at the stage of creating super-efficient neural networks. For me, this is not quite the first stage, first I had to understand the principles of the peripheral nervous system and how neural networks work in simple nervous systems and, in general, create more than one model of the operation of neurons. Now I am developing a model for the main structural element, which is not a neuron, but a hypercolumn or cortical module. Hence the working name hypercolumn network or hypernet for short.

Next, I will tell you in more detail what principles it is based on and how I implement it.

Second signaling system — nodes

Let’s see how information from receptors can be represented in the cortex, for this, we need to understand how two simple mechanisms of irradiation and induction, known for over a century, proceed.

Irradiation is the tendency of nerve excitation to spread to neighboring neurons or neighboring nerve tissue.

Usually, the processes of cells of one column can spread to adjacent columns and exert a stimulating effect in adjacent columns. The nature of such an incentive effect on adjacent columns can be represented as a certain circular halo with a maximum effect in the center and with a gradual weakening with distance from the center.

Irradiation map

Induction is the appearance of areas of inhibition on the cortex around the site of excitation. This may be because outgrowths of neurons of one column at a certain distance of the column can form connections with inhibitory cells of other columns. The nature of the inhibitory effect on adjacent columns can be represented as follows:

Induction map

These processes work in the cerebral cortex at the same time. Let’s simulate how the excitation arising in the cortex under the influence of these two mechanisms will transform. For this, a pixel shader was written — a program executed by the GPU for each pixel in an image.

Shader in Unity

In the model, each pixel of the image will be a separate cortical module; the brighter the pixel, the stronger the overall excitation in the column.

An example of the shader, the image size is 28x28 pixels, the radius of the column is 5 pixels, the shader is applied several times

Column radius of 7 pixels

As you can see, under the action of the mechanisms of irradiation and induction, any activity is broken down into small foci of excitation. The size of these foci depends on the size of the total impact on neighboring hypercolumns, as well as on the ratio of the forces of action of incentive and inhibitory maps.

These foci of excitation are encapsulated, as it were, surrounding themselves with an area of ​​inhibitory action. Also, their central part, with several applications of shader calculations to the image, due to irradiation, becomes more and more active, thereby these foci become pronounced and stable.

These centers of excitation will hereinafter be called nodes. Nodes are some new emergent property of the cortex; it manifests itself as a result of the mutual work of two mechanisms of irradiation and induction. The nodes are not tied to the size and position of either neurons or hypercolumns, they work as if on these structures. The combination and location of nodes is the representation of information in the cerebral cortex.

Nodes are the second signaling system, symbolic representations of images in the brain.

It is amazing how simple and elegant solutions are used by the cortex and the brain as a whole.

To understand how nodes can store images or parts of images, it is necessary to study another mechanism at work in the cortex.

Reverberation of excitation

The reverberation of excitation is a prolonged circulation of excitation impulses in networks of neurons or between different structures of the central nervous system. For us, corticothalamic reverberations and reverberations that occur between different areas of the cortex are of great importance. It is these reverberations that create various patterns and rhythms in the electroencephalography of the brain. For such reverberations, descending or feedbacks is required, which are usually several times more than ascending direct connections.

We need four maps to simulate the reverb process. Here I refer to the images as maps, to which we will apply some pixel shaders, if necessary for our model.

A map of the thalamic nucleus onto which we will project some images, in this case, examples of handwritten numbers from the standard MNIST set. The images in the MNIST dataset are 28x28 pixels, so we’ll set this map to 28x28 pixels as well.

The cortex map will be 21x21 pixels in this version.

The map of ascending synaptic connections will store the values ​​of all weights of the synapses connecting the thalamic nucleus map and the cortex map. Conventionally, each pixel-hypercolumn of the cortex will have synapses with 64 (8x8 nucleus) pixels of the thalamic nucleus map. If you move the 8x8 pixel kernel along with the thalamic nucleus map in one-pixel increments, then you can make a map of the ascending synaptic connections. Accordingly, this gives the size of the cortex map 28–8 + 1 = 21, in turn, the size of the map of ascending synaptic contacts will be 168x168 pixels (21 x 8 = 168).

The synapse weight will be determined by a number from 0 to 1 — this is convenient for graphical display on a map and allows you to store and process these parameters in the form of images, which is also convenient in working with shaders.

Another important point, in the models that I will describe, there are always two maps of synaptic connections: one map of incentive synapses, the second map of inhibitory synapses. These are always two cards of the same size.

It is known that the type of synaptic contact in the biological nervous system does not change during life, that is, an incentive synapse cannot become inhibitory, or vice versa. In computer neural networks, the sign of the synapse weight can easily change during the learning process, which is not correct for a model claiming to be biological.

In the columns of the cortex, in addition to the incentive pyramidal neurons, there are many inhibitory neurons, and any axon from the incentive cell directed to the column can create synaptic contact with both the incentive and inhibitory cells. Similarly, we have two synaptic contact cards, one will act with a plus sign, the other with a minus sign. In both of these cards, the weight will be determined by a number from 0 to 1.

We fill the value of the weights of the ascending synaptic contacts with random numbers from 0 to 1.

The map of descending synaptic connections will be significantly larger. As already noted, there are several times more descending connections in the biological brain. In our case, we choose a map size that will demonstrate the concept of storing images in nodes. Each pixel-hypercolumn of the cortex map will be associated with each pixel of the thalamic nucleus map, that is, with every 784 pixels (28 x 28 = 784), then the map size will be 588x588 pixels (21 x 28 = 588).

Initially, the value of the weights in the map of descending synaptic connections will all be equal to 0, but we will change these weights during training.

The teaching method that we in this case apply to the nervous system has been experimentally confirmed, this is the Hebb rule. Its simplest interpretation: we work together, it means we strengthen the connection; as well as the downside: if they didn’t work together, the connection is weakening.

First, let’s give our model an example by projecting it onto a map of the thalamic nucleus. Then, through the upward connections, we have a summing effect on the map of the cortex. For each pixel of the cortex map, all the corresponding weights from the map of ascending synaptic connections are added, multiplied by the value of the activity level of the corresponding pixels of the thalamic nucleus map. The value of the weights from the inhibitory map of the ascending synaptic connections will be added with a minus sign. The resulting amount is multiplied by the selected coefficient, called the sensitivity coefficient. In this case, it is 0.5.

If the received sum is less than or equal to zero, then the pixel-hypercolumn is not activated, its activity is equal to zero. If the resulting sum is greater than one, then the pixel value of the bark map will be equal to one, like the maximum degree of activation of the hypercolumn. If the sum has a value in the range from 0 to 1, then the pixel-hypercolumn of the bark map will take that value.

The sensitivity coefficient is selected so that as a result of the sum, a very large number of hypercolumns of the cortex map is not activated since this leads to mutual suppression of activity when using induction. Also, the sensitivity factor should not be too low, because this can lead to the very low activity of hypercolumns.