SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hao Zheng† Hui Lin† Rong Zhao*
Center for Brain-Inspired Computing Research,
Optical Memory National Engineering Research Center,
Tsinghua University - China Electronics Technology HIK Group
Co. Joint Research Center for Brain-Inspired Computing,
IDG / McGovern Institute for Brain Research at Tsinghua University,
Department of Precision Instrument,
Tsinghua University, Beijing 100084, China.
{zheng-h17, linhui21}@mails.tsinghua.edu.cn
r_zhao@mail.tsinghua.edu.cn
Dynamically grouping sensory information into structured entities is essential for understanding the world of combinatorial nature. However, the grouping ability and therefore combinatorial generalization are still challenging artificial neural networks. Inspired by the evidence that successful grouping is indicated by neuronal coherence in the human brain, we introduce GUST (Grouping Unsupervisely by Spike Timing network), an iterative network architecture with biological constraints to bias the network towards a dynamical state of neuronal coherence that softly reflects the grouping information in the temporal structure of its spiking activity. We evaluate and analyze the model on synthetic datasets. Interestingly, the segregation ability is directly learned from superimposed stimuli with a succinct unsupervised objective. Two learning stages are present, from coarsely perceiving global features to additionally capturing local features. Further, the learned building blocks are systematically composed to represent novel scenes in a bio-plausible manner.
Humans are able to effortlessly group the world in terms of abstract and reusable 'building blocks', and leverage those learned blocks to understand novel contexts. For example, there is a list of evidences that infants as young as 3-month old are able to group information from individual visual elements into wholistic perceptual units [1, 2, 3, 4]. This phenomenon is referred as perceptual grouping [5] or binding in neuroscience and psychology literature [6] and such ability is believed to be fundamental for the compositional generalization of neural network models [7].
However, grouping ability challenges artificial neural networks (ANNs) for several reasons. Firstly, grouping from raw sensory input is in principle an unsupervised task with multiple possible solutions. For example, how to segment a scene[8] or how to chunk a sensory stream into episodic memories[9] do not have a golden criterion. This excludes the solution through massive labeling and supervised learning[10]. Secondly, the representation of symbol-like content (the pattern of a single object) should be able to retain itself and be reused in front of various situations with a common format. Without proper constraints, simple network models can fall into ambiguities when trying to compose
$\dagger$  Equal contribution.
The code is publicly available at: https://github.com/monstersecond/gust.
those learned patterns, referred to as the binding problem[11]. Thirdly, the cost of representing all possible combinations statically and locally is exponentially high, limiting the capability of the local code to deal with increasingly complex scenes[12, 13]. Lastly, the grouping ability should not be constrained to an explicit prior of separation [14], but can organize itself flexibly and dynamically according to the statistical structure of the input.
In this paper, we take insights from the neural system of the human brain to provide a brain-inspired grouping mechanism that mitigates the aforementioned challenges in ANNs. Neurons in the brain have a spiking nature. The firing activity in the high-dimensional neuronal space can have both spatial and temporal structures. The spatial structure of spike firing, usually referred to as receptive field, is related to certain objects[15] or certain feature content of objects (orientation, color, texture, depth etc.)[16]. Occasionally, the neuronal firings evolve into a coherent state, where large amounts of neurons interrelate in the millisecond timescale (Fig.1b,right)[17]. Specifically, neurons encoding features of the same entity tend to fire synchronously and out of phase with neurons that encode features of different entities[18]. At this time, the temporal structure of the spike firing emerges as the neuronal oscillation arises[19, 20]. In the cognition literature, this coherent state is highly related to awareness and successful percepts [21, 22, 23, 24]. According to these evidences, humans appear to use the spatial and temporal structure of spikes for representing contents and the grouping information among contents, respectively. Salient coherent states are indicators of the successful grouping, which enable better readout of contents by downstream neurons with non-linear dendrites sensitive to coincident events[25]. These insights have been developed as temporal binding theory[26, 27] or correlation brain theory[6] in the neuroscience literature.
Compared with grouping mechanisms in ANNs, grouping with temporal neuronal coherence has several promising advantages. First, the grouping is a flexible and dynamical self-organized process without supervision or explicit prior of the separation. Second, the coherence level may help to encode the grouping uncertainty. Third, grouping in time dimension will naturally maintain a common format[7]. Lastly, the (spatial) representational cost is reduced to a single pool of perceptual elements[7]. However, despite the advantages of grouping with temporal coherence, as far as we know, such grouping mechanism is seldom present in ANNs field due to the gap between temporal binding models and gradient-based ANNs. To be more specific, precise temporal codes rely on spiking neurons, which are non-differentiable and also require simulating the dynamics of neurons. Besides, the connection between neurons is required to be built-up by hebbian-like plasticity rule[28] instead of gradient-based learning rules[29], which necessitates a completely different training framework from that of ANNs. Lastly, it is not clear how neuronal coherence emerge within ANNs framework.
In this paper, we introduce the GUST network (short for Grouping Unsupervisely by Spike Timing network) that assigns the grouping information implicitly in spiking synchrony as an emergent property of the network. The coherent state is achieved by iterative bottom-up / top-down processing between a spike coding space (SCS) biased by biological constraints and a denoising autoencoder (DAE)[30, 31] to provide delayed feedback. The top-down delayed feedback architecture for the temporal grouping is closely related to the constructive nature of the brain perception [32]. Notably, the GUST is trained end-to-end by a gradient-based method and therefore compatible with ANNs framework. Besides, the grouping ability is learned implicitly with a general unsupervised objective to directly denoise multi-object visual inputs. Lastly, once the GUST "sees" a given number of objects during training, the acquired grouping ability can be successfully generalized to visual inputs of a different number of objects, showing its combinatorial generalization capability to understand scenes of different structures (eg. number of objects).
Designing GUST is non-trivial for four challenges. First, stochastic dynamics based on spike firing is non-differentiable and gradients cannot be naturally back-propagated as standard artificial networks. Additional biological constraints like refractoriness, non-linear dendrites, transmission delay and timescale difference aggravate the problem. Second, temporal grouping requires relatively high temporal resolution, which in turn implies a long simulation period. Such long simulations challenge the efficiency of backpropagation. Third, as far as we know, there are no trivial unsupervised objective functions to train a temporal grouping network directly from superimposed inputs. Lastly, the representation needs to be explainable in terms of its precise temporal structure. To overcome challenges for back-propagation, we approximate gradients in non-differentiable parts and short-cut the back-propagation by the transmission delay (Section3.5). For unsupervised learning, surprisingly, a super succinct objective function to denoise superimposed visual inputs is shown to enable the GUST to learn to segregate (Section3.4), once biased by a list of biological constraints (eq.1~eq.7).
For evaluation, we develop methods to quantitatively evaluate neuronal coherence and its relation to the grouping (Section3.6) based on a non-euclidean metric of the precise temporal coding in the visual cortex[33]. By quantitative evaluation and qualitative visualization, the representation in GUST is clearly explained.
In sum, the contribution of GUST is threefold. First, a gradient-based, unsupervised training framework is developed to enable the GUST to group objects with neuronal coherence. Since the framework is compatible with ANNs, it contributes to bridging the temporal binding theory and the ANNs framework. Second, the GUST is the first temporal grouping model that learns to group directly from multi-object inputs and is able to systematically generalize the mechanism to inputs of different number of objects. Third, we develop a clustering method to evaluate the neuronal coherence through a non-linear metric that measures the distance between arbitrary spike trains directly based on their precise temporal structures.
2 Grouping in the temporal dimension
Grouping. In this work, grouping is formalized as a pixel-level segmentation of visual scenes composed of multiple objects (Fig.1a). It is notable that while only visual inputs are used for clarity, the general mechanism is not restricted to a certain modality. Similarly, if we take the pixel-level elements as more abstract features that are distributed in a high-dimensional representation space, the grouping mechanism is not restricted to the superficial features as well.
Grouping with neuronal coherence. Two separated timescales are considered:  $\tau_{1} \ll \tau_{2}$  (Fig.1b). During a period of  $\tau_{2}$ , we assume that the accumulated firing rate of each neuron reflects certain features of an object. In other words, the spatial structure of a population of neurons encode which feature are present in the current scene. In this work, features are formalized as object-related pixels in an image. When the neuronal coherence state is arrived, the firing of neurons is synchronized within a much narrower temporal window  $\tau_{1}$  so that the tempo
ral structure of spike firing emerges (Fig.1b,right). The temporal structure is reflected in the regular rhythmic pattern of the population activity (LFP), which act as an emergent coordinate  $(z_{i},z_{i + 1},z_{i + 2}$  in Fig.1b ) to assign the additional grouping information. It is notable that the temporal coordinate is not an explicit prior from human experts but an emergent property from the evolution of the network dynamics, Therefore, the grouping is flexible and dynamic.
Combinatorial generalization by grouping. If we formalize "seeing" a scene as successfully grouping the visual objects with neuronal coherence, the combinatorial generalization of the grouping is in turn defined as the following question: Given that the GUST has only "seen" scenes composed of a certain number of objects during training, does it generalize its grouping ability to "see" scenes composed of a different number of objects, which structurally change the input distribution(Fig.1c)? In other words, can GUST form coherent neuronal groups even in unseen situations? This question requires the GUST not only learns to group certain types of scenes into building blocks, but also manages to compose the building blocks to understand novel scenes.
Figure 1: (a) Grouping as segmentation. (b) Grouping with neuronal coherence. The spike plot shows the emergence of the neuronal coherence (x-axis: time; y-axis: neuronal index). (c) The task scheme for combinatorial generalization
Figure 2: (a) The spiking neuron model. (b) The general architecture of GUST. (c) The unrolling of GUST through time.
In this section, we first introduce the general architecture of GUST (Section 3.1) and provide details of each module (Section 3.2,3.3). Then, we introduce how three essential challenges are solved including designing the unsupervised objective function (Section 3.4), back-propagation strategy to train the GUST (Section 3.5) and how grouping with neuronal coherence is evaluated (Section 3.6).
3.1 General architecture of GUST
How can the coherent states emerge from random firing? While it is still an intricate question without a unified answer, we take inspiration from the anticipating nature of the neural activity in a processing hierarchy. The top-down feedback to the primary sensory cortices carries predictions about the features of the environmental stimuli and such modulatory top-down effects might influence the temporal structure or coherence level of the neuronal responses[32, 34, 35]. Therefore, the predictive feedback acts as positive cues to guide the grouping and coherence and is realized as the reconstructive output of an autoencoder in GUST. Besides, spiking dynamics, as pointed out in neuronal sampling theory[36], provides biological constraints that facilitate bypassing the energy barrier to switch between alternative stable states. In this way, the contents of different groups can get separated in time (See Appendix A.3.1).
The general architecture of GUST is composed of a spike coding space (SCS, green square in Fig.2b) and a denoising autoencoder (DAE, blue shadow in Fig.2b). The grouping solution is represented as neuronal coherence in the SCS and is found by iterative bottom-up neuronal sampling in the SCS and top-down attentional feedback  $\gamma$  from the DAE during the whole simulation period  $T$ . The feedback is delayed by  $d$  steps.
3.2 Spike coding space (SCS)
The SCS is a layer of the same dimension as input  $x$  and is composed of two-compartment pyramidal neuron models (Fig.2a). Each pyramidal neuron receives the driving signals from the noisy visual inputs  $\tilde{x}$  (during training*) and the delayed modulatory feedback from the DAE output.
$$
\gamma_ {i} ^ {\prime} (t) = \gamma_ {i} (t - d). \tag {1}
$$
where  $\gamma_i^\prime$  is the delayed modulatory feedback to neuron  $i$ ,  $\gamma_{i}$  is the output of DAE and  $d$  is the delay period. Two sources of inputs are multiplied to determine the noiseless membrane potential  $\rho_{i}(t)$ :
$$
\rho_ {i} (t) = \gamma_ {i} ^ {\prime} (t) \cdot \tilde {x} _ {i}. \tag {2}
$$
The stochastic neuron fires a spike if it is not refractory and its membrane potential exceeds a given threshold. More formally,
$$
s _ {i} (t) = H \left(\rho_ {i} (t) + \epsilon ; \delta_ {i} (t)\right). \tag {3}
$$
where  $\epsilon$  is the random noise drawn from a uniform distribution  $U(0,1)$  and  $H$  is the non-linear activation function of the spiking neurons (threshold  $= 1$ ):
$$
H \left(u _ {i}; \delta_ {i}\right) = \left\{ \begin{array}{l l} 1, & \delta_ {i} = 0 \wedge u _ {i} \geq 1. \\ 0, & \delta_ {i} = 0 \wedge u _ {i} <   1. \\ 0, & \delta_ {i} > 0. \end{array} \right. \tag {4}
$$
*During testing and visualization, it receives original inputs  $x$ , see Appendix A.7 (Salt&Pepper noise).
where  $\delta_i(t)$  is the refractory variable for each neuron, which is non-zero if the neuron is refractory. A neuron falls into refractory period of length  $\delta$  if it fires a spike. More formally,
$$
\delta_ {i} (t) = \left\{ \begin{array}{l l} 0, & s _ {i} (t) = 0 \wedge \delta_ {i} (t - 1) = 0. \\ \delta_ {i} (t - 1) - 1, & s _ {i} (t) = 0 \wedge \delta_ {i} (t - 1) > 0. \\ \delta , & s _ {i} (t) = 1. \end{array} \right. \tag {5}
$$
To sum up, the SCS integrates the driving input and the delayed modulatory feedback multiplicatively to determine the spike firing of neurons that are not in the refractory period. Once a neuron fires a spike, it gets into the refractory period that is unable to give another spike firing. The refractory dynamics provides the essential temporal competition to separate groups of different objects and acts as a structural bias for a grouping solution[37].
3.3 Denoising autoencoder (DAE)
The coherent activities in SCS are readout by coincident detectors (CD) that integrate adjacent activities within a very narrow time window  $(\tau_w = 2)$  to feed to subsequent bottom-up processing.
$$
\begin{array}{l} s _ {i} ^ {\prime} (t) = C D \left(s _ {i} (t), \dots , s _ {i} \left(t - \tau_ {w} + 1\right)\right) \\ = \left\{ \begin{array}{l} 1, \quad \sum_ {t ^ {\prime} = 0} ^ {\tau_ {w} - 1} s _ {i} \left(t - t ^ {\prime}\right) \geq 1. \\ 0, \quad \sum_ {t ^ {\prime} = 0} ^ {\tau_ {w} - 1} s _ {i} \left(t - t ^ {\prime}\right) <   1. \end{array} \right. \tag {6} \\ \end{array}
$$
The bottom-up and top-down communication in a processing hierarchical[32] is formulated as a denoising autoencoder that is composed of an encoder  $f$ , a latent layer  $\theta$  and a decoder  $g$  (Fig.2b). The encoder extracts more abstract features into the latent representation  $\theta$  and the decoder computes the top-down feedback  $\gamma$  to SCS. More formally,
$$
\gamma_ {i} (t) = g \left(f \left(s _ {i} ^ {\prime} (t)\right)\right). \tag {7}
$$
The dimension of input and output of DAE are kept the same as SCS and the dimension of the latent space is much smaller to form an information bottle neck[38] of the hierarchical processing.
The first challenge is how grouping with neuronal coherence can be learned in an unsupervised way similar to young infants. For this motivation, the objective function should not leak any explicit grouping information.
We train the GUST to actively predict and denoise the multi-object input patterns at timescale  $\tau_{2}$ . Only a simple unsupervised loss function is identified:
$$
L (x) = \left(x - \frac {1}{Z} \sum_ {t = T - \tau_ {2}} ^ {T} \gamma (t)\right) ^ {2}. \tag {8}
$$
$L(x)$  is the MSE distance between the original inputs  $x$  and the averaged denoising predictions by the DAE (averaged over  $\tau_{2}$  time steps).  $T$  is the simulation length and we select the last segment of length  $\tau_{2}$  where it is more possible to reach a convergent state (Fig.2c).  $Z$  is a normalization factor.
Notably, the objective of the training process is consistent with the constructive nature of the perception and only related to multi-object inputs as well as the firing rate of neurons. Therefore, the grouping of objects and the neuronal synchrony are not explicitly guided by the objective function itself. Instead, they arise "effortlessly" as an emergent phenomenon along with anticipating the world[32], like an infant[4].
Figure 3: The illustrative scheme of SynClu. Gray box illustrates how VP metric  $(D_{VP})$  is computed.  $s_{i / j}$  is the spike train of neuron  $i / j$ .  $T$  is an arbitrary list of transformation to transform  $s_i$  to  $s_j$ .  $T(u)\in \{\mathrm{del,add,shift}\}$ .  $D_{VP}$  is defined as the minimal cost to transform one spike train  $s_i$  to another  $s_j$ . The right side shows how SynClu cluster the neurons (exemplified for a single clustering iteration). Step I: from clustering assignment  $I(c_k)$  to updated k cluster centers  $s_i^{c_k}$ .  $I(c_k)$  is the set of spike trains in the cluster  $c_k$ . Step II: from k clustering centers to updated cluster assignment  $C(s_i)$ .
3.5 Gredients and unrolling
The second challenge is how to train the GUST in a gradient-based manner, given that the end-to-end training of the GUST is not naturally compatible with the standard deep learning framework due to the binary and dynamical nature of the SCS.
The challenge comes from two parts. One is the non-differential elements in SCS, like spiking non-linearity (H, eq.4), coincidence detector (CD, eq.6) and refractoriness  $(\delta_{i}$ , eq.5). Another is the long simulation period  $(>100)$ , which makes it harder to backpropagate the gradients due to stability issues [39]. For the spiking non-linearity we approximate the Heavistep function H as a piece-wise linear function to compute a surrogate gradient[40, 41]. The same strategy is used to pass the gradient through the threshold of coincidence detectors. As for refractoriness, we cut off the gradient through the refractory variable  $\delta_{i}$  to simplify the computation graph. As a result, the refractory variable acts as a gating variable so that the gradients can only be passed through neurons that are not in their refractory periods in the unrolling graph (green dots in the SCS in Fig.2c). Lastly, the transmission delay naturally shortcuts the depth of the backpropagation for  $d$  times so that the backpropagation of errors along the simulation period is much more efficient (green lines in Fig.2c). The green lines and green dots in Fig.2 (c) stand for the shortcuts or neurons where the gradients can be back-propagated.
3.6 Evaluation
The last challenge is how to quantitatively measure the neuronal coherence in a high dimensional spike coding space and evaluate the grouping performance based on neuronal synchrony.
The evaluation of the coherent neuronal groups requires a clustering method directly based on the temporal structure of the spike firings. Therefore, we develop SynClu (Fig.3), a clustering method that combines a spike-timing-based metric and K-medoids clustering together. The Victor-Purpura metric  $(D_{VP})$  that we use is a classical non-Euclidean metric for analyzing the precise temporal coding in the visual cortex[33]. The K-medoids[42] thereby iteratively assign K clustering centers  $(s_i^c)$  among given neurons  $(s_i)$  to minimize the inner-cluster VP distance. The convergent clustering assignment is compared with the ground truth grouping by the adjusted mutual information (AMI)[43] to evaluate the grouping quality (1 is perfect grouping while 0 is the chance level). The Silhouette score[44] indicates the inner-cluster coherence level of the spiking pattern, therefore is denoted as the synchrony score (SynScore).
4 Experiments and results
Since the grouping is formalized as a pixel-level segmentation of visual scenes composed of multiple objects (Section.2 Grouping), synthetic images composed of multiple Shapes[45] are generated for evaluation. By generating a different number of shapes (Fig.1bottom, Section.2 Combinatorial generalization by grouping), we can control the statistical structure of the training data and testing data, as well as access to the ground truth of grouping assignment. This allows us to evaluate the grouping performance and further evaluate the combinatorial generalization ability of the approach.
In the following sections, we unfold the grouping property of GUST step by step. We first demonstrate the emergence of neuronal coherence during the simulation of a well-trained GUST, which softly indicates the grouping assignments (Section.4.1). To figure out why the intriguing grouping capability is learned without explicit supervision, we dive into the details of the training process. Notably, it is shown that the DAE of GUST gradually learns to denoise single objects instead of multiple objects (Section.4.2). Further visualization of the output attention map  $\gamma$  shows that the DAE learns in a phasic manner, from coarsely predicting the global features (position) to additionally capturing local features (shapes) of single objects, which may provide a heuristic explanation of the mechanism. Lastly, given that GUST can learn to segregate multi-object scenes into grouped single-object representations during training, we study whether the single-object representation can be composed to understand novel scenes with more or fewer objects than ever encountered during training (Section.4.3). In other words, we ask whether the GUST generalizes its grouping mechanism in a combinatorial manner. For visualization and analysis in diverse cases, see Appendix A.9  $\sim$  A.11.
4.1 Emergence of neuronal coherence for grouping