# Classification of Point Clouds with Neural Networks and Continuum-Type Memories

Stefan Reitmann, Elena V. Kudryashova, Bernhard Jung, Volker Reitmann

► To cite this version:

Stefan Reitmann, Elena V. Kudryashova, Bernhard Jung, Volker Reitmann. Classification of Point Clouds with Neural Networks and Continuum-Type Memories. 17th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2021, Hersonissos, Crete, Greece. pp.505-517, 10.1007/978-3-030-79150-6_40. hal-03287683

HAL Id: hal-03287683

https://inria.hal.science/hal-03287683

Submitted on 15 Jul 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Classification of Point Clouds with Neural Networks and Continuum-Type Memories

Stefan Reitmann¹[0000-0003-0283-8272], Elena V. Kudryashova², Bernhard Jung¹, and Volker Reitmann²

¹ Freiberg University of Mining and Technology, Institute of Computer Science, Chair of Virtual Reality and Multimedia, Germany
stefan.reitmann@informatik.tu-freiberg.de

² St. Petersburg State University, Department of Mathematics and Mechanics, Chair of Applied Cybernetics, Russia
e.kudryashova@spbu.ru

**Abstract.** This paper deals with the issue of evaluating and analyzing geometric point sets in three-dimensional space. Point sets or point clouds are often the product of 3D scanners and depth sensors, which are used in the field of autonomous movement for robots and vehicles. Therefore, for the classification of point sets within an active motion, not fully generated point clouds can be used, but knowledge can be extracted from the raw impulses of the respective time points. Attractors consisting of a continuum of stationary states and hysteretic memories can be used to couple multiple inputs over time given non-independent output quantities of a classifier and applied to suitable neural networks. In this paper, we show a way to assign input point clouds to sets of classes using hysteretic memories, which are transferable to neural networks.

**Keywords:** Point Clouds · Classification · Neural Network · Hysteretic Memory

## 1 Introduction

Depth sensors have become ubiquitous in many application areas, e.g. robotics, driver assistance systems, geo modeling, and 3D scanning using smartphones. Depth sensing is achieved by means of waves or rays that are sent out by a transmitter, reflected at surfaces and detected again by a receiver. The time difference between emitting and receiving gives information about the distance covered. Various physical principles are used, such as electromagnetic waves (radar), acoustic waves (sonar) or laser beams (LiDAR). In our research, we aim at developing an autonomous robotic platform that serves as a carrier for various LiDAR and sonar sensors for 3D mapping and environmental data acquisition in inland waters [2]. 3D point clouds obtained from ultrasonic and LiDAR mapping need to get semantically segmented and classified through machine learning analysis to extract knowledge for object detection and autonomous movement. In order to train such classifiers, however, large amounts of training data are required that provide labeled examples of correct classifications. Therefore we developed an approach where virtual worlds with virtual depth sensors are used to generate labeled point clouds (fig. 1 [16]). To prepare an application in certain use cases, AI
algorithms (e.g. neural networks) are trained with synthetic data obtained in virtual environments under various conditions and signal qualities [16].

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Exemplary synthetic labeled point cloud of virtual LiDAR sensor with ground truth data for part segmentation of 3D chair meshes. The axes represent independent units of measurement of the virtual environment (Blender 3D software suite).</figcaption></figure>

However, a distinction must be made here between a completely registered point cloud (as depicted in fig. 1) and raw temporal impulses from the sensors. For active use in robotic movement and automization, the raw temporal impulses must be evaluated, while the completely registered point cloud is created in postprocessing, but initially offers no added value for the motion behavior. Therefore, ways have to be found to extract knowledge for object recognition and classification from the raw temporal impulses, as these are the data the robot gets at a certain point of time.

This approach differs from a variety of considerations on how machine learning can be used to classify point clouds (see sec 2.2). In order to use temporal impulses for classification in a meaningful way, it must be possible to link several impulses of different points of time with each other. Accordingly, it must be possible to link historical output variables with each other via a memory. The classifier is thus not solely dependent on the input variables (sensor pulses), but also on the previous state of the output variable. The system can thus - depending on the previous history - assume one of several possible states for the same input variable. These considerations lead us to the assumption that hysteresis memories and neural network applications can be coupled with each other.

## 2 Background

This section details the fundamental concepts of light and sound propagation and reflection under ideal and perturbed conditions.
## 2.1 Depth Sensing

LiDAR & Time-of-Flight (ToF) LiDAR is a common method for optical distance measurement. Generally, passive and active sensor systems can be distinguished. In this article we focus on active distance measurement, where radiation is introduced into the environment by the measuring device. An example are cameras that can determine distances by means of ToF or the active triangulation [11].

Sonar / Ultrasonic Sonar is a distance measurement method based on sound waves. Like LiDAR, measurements devices can be divided into active and passive<fnref n="3" /> Sonar. Again, only the more common active type is considered in this paper. Here, the transmitter emits a signal in the form of a sound wave. The sound wave is reflected at the target object and registered at the receiver. The time difference between signal transmission and reception provides information about the distance. The type can be one of the options rotating, static, or side-scan. Depending on the sensor, different configuration options are available for the sensor's field of view. The first two sensors have a field of view in horizontal and vertical direction, while the side-scan sensor only has a downward opening angle. Active sonars are used, for example, to locate schools of fish in waters or to map underwater structures (see fig. 2).

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Temporal pulses of side scan sonar in virtual underwater environment. Red points are grouped to the underground, blue points to irregularities, e. g. certain underwater objects. Each segment compries one sonar pulse at one certain point of time. Again the axes represent independent units of measurement of the virtual environment (Blender).</figcaption></figure>

The task is now to be able to correctly assign the points to object classes with an arbitrary classification function $f$. However, since there is no complete point cloud, but only individual pulses, the object classes must be detected via the connection of several

<footnote n="3">³ In passive Sonar, the target object itself rather than the sensing device emits a sound signal. This signal can be identified by its characteristic signal profile.</footnote>
time steps. This approach has different requirements than those found, for example, in
[13]. In the following section, we therefore discuss previous approaches and work out
the requirements for the model.

## 2.2 Classification and Semantic Segmentation

Following the assumptions of [14], we can summarize the main properties of the input
point clouds or a subset of them:

- Unordered: points in a point cloud are typically not assumed to have any particular structure

- Density variability: non-constant density within the point clouds due to perspective effects, movements or measurement errors

- Invariance under transformation: as a set, such data must be invariant to mutations of its members

One option for automated processing of point clouds with machine learning, e. g. for segmentation, is unsupervised learning. For example, a cluster analysis can be used to segment a point cloud into certain parts. Such a procedure is described in [4] and [3]. The classification of data points into groups (also: clusters) is done by grouping elements that are as similar as possible to each other. More detailed information on this can be found in [1].

Supervised learning is usually used for classification tasks and, in contrast to un-
supervised learning, requires semantically labeled examples for training. For 2D image
data, the learning of classifies from examples is described e.g. or [15]. Learning of clas-
sifiers for the case of three-dimensional data has been investigated e.g. in [13], [14] and
[24]. Regardless of the number of dimensions, methods of supervised learning are often
used to automatically recognize patterns and relationships. Such an approach, however,
requires a large amount of training data with the correct classification for each pixel or
point in a scene (pixel-wise or point-wise segmentation). Alternatively, smallest circum-
scribing rectangles or bounding boxes may be used depending on the needed amount of
precision.

So various applications have shown that the use of neural networks in the analy-
sis of LiDAR and sonar data yields promising results for classification and semantic
segmentation. However, for most of the methods a pre-processing of the pulse signals
derived from LiDAR sensors is necessary, which results in high computational overhead
and considerable latency. Spiking Neural Networks (SNN), referred as third generation
neural networks, showed promising results in modeling time dependent data as depth
sensing signals (see 2.1) are. SNN are capable to compute the raw temporal impulses of
LiDAR and sonar and are so meaningful in the real-time automization of robotic move-
ments. Using the synthetic raw pulse data of simulated depth sensors derived in the
virtual environments (like in fig. 2), we want to address the object recognition problem
with SNN on raw temporal pulses.

However, both conventional Deep Learning structures and SNN suffer from a non-
transparent representation from an input set to an output class. To extend these capabil-
ities, hysteresis memories can be used, which can address the described classification
problem through convergence behavior.
# 3 Continuum stationary sets in an acoustic-ODE problem

## 3.1 Stability investigations by abstract Volterra integral equations

In this section we consider a boundary control problem for the interactions of acoustic waves with some control processes on the boundary described by an ODE. Note that on the level of time-series this coupling problem can arise as temporal coding in SNN [6]. Our aim is to provide frequency-domain conditions for the convergence of trajectories of the process to the stationary set (attractor) which consists of a continuum of functions. In order to show this we introduce some function spaces needed in the sequel and consider a general Volterra integral equation. The problem of convergence of solutions of the PDE/ODE problem will be reduced to the investigation of such an integral equation.

The interaction of acoustic and laser signals with a structure (control or heating part) can be described as hybrid system consisting of different types of PDE's and ODE's connected by special boundary conditions. In the case of a microwave heating problem the incoming signal is given by Maxwell's equations, the heating process is defined by parabolic equations. Under certain assumptions in this situation a multivalued dynamical system having a global attractor is generated [19].

For a Hilbert space $Y$ with scalar product $(\cdot, \cdot)$ and norm $|\cdot|$ the space $L_{\text{loc}}^2(\mathbb{R}; Y)$ consists of locally $L^2$-functions on $\mathbb{R}$ with values in $Y$ and with a topology defined by the family of seminorms $|y|_n := (\int_{-n}^n |y(t)|^2 dt)^{1/2}$, $n = 1, 2, \dots$. Thus the space $L_{\text{loc}}^2(\mathbb{R}; Y)$ is considered as a Fréchet space, i.e. as a complete metrizable linear topological space. For any interval $J \subset \mathbb{R}$ we regard $L_{\text{loc}}^2(J; \mathcal{Y})$ as a subspace of $L_{\text{loc}}^2(\mathbb{R}; Y)$ identifying $L_{\text{loc}}^2(J; Y)$ with the set of functions in $L_{\text{loc}}^2(\mathbb{R}; Y)$ which vanish outside of $J$. For any interval $J \subset \mathbb{R}$, a Hilbert space $Y$ and a parameter $\rho \in \mathbb{R}$ we introduce the weighted spaces $L_{\rho}^2(J; Y)$ and $W_{\rho}^{1,2}(J; Y)$ by

$$L_{\rho}^{2}(J; Y) := \{f \in L_{\text{loc}}^{2}(J; Y) \mid \int_{J} e^{-2\rho t} |f(t)|^{2} dt < \infty\}$$

$$\text{and} \quad W_{\rho}^{1,2}(J; Y) := \{f \in L_{\rho}^{2}(J; Y) \mid \dot{f} \in L_{\rho}^{2}(J; Y)\}.$$

$(\dot{f}$ denotes the distribution derivative.)

Assume that the map $t \in \mathbb{R}_+ \mapsto \mathcal{L}(U, Z)$ is twice piecewise-differentiable and satisfies the following condition: There exists a $\rho_0 > 0$ and a constant $\gamma > 0$ such that

$$\| K(t) \|_{\mathcal{L}(U,Z)} \le \gamma e^{-\rho_0 t}, \quad \forall t > 0, \qquad (3.1)$$

$$\text{and} \quad \int_{0}^{\infty} [ \| \dot{K}(t) \|_{\mathcal{L}(U,Z)}^{2} + \| \ddot{K}(t) \|_{\mathcal{L}(U,Z)}^{2} ] e^{2\rho_{0}t} dt < \infty. \qquad (3.2)$$

Consider the Volterra integral equation

$$z(t) = h(t) + \int_{0}^{t} K(t - \tau) \phi(z(\tau), \tau) d\tau, \qquad (3.3)$$

where $K(t) \in \mathcal{L}(U, Z)(U, Z)$ Hilbert spaces) is twice piecewise-differentiable satisfies (3.1) and (3.2), and has therefore a state-space realization. Suppose that $\phi: Z \times \mathbb{R}_+ \to U$ is a continuous function.
Instead of one fixed nonlinearity $\phi$ we consider a family $\mathcal{N}$ of continuous maps, such that for any $\phi \in \mathcal{N}$ and any $h \in \mathcal{D}(A)$ with $\mathcal{D}(A)$ from the nonlinear integral equation (3.3) has a unique solution $z(\cdot, h, \phi)$ and this solution is continuous.

In order to describe the absolute stability or instability behavior of (3.3) in the following theorem we need an additional assumption on the class $\mathcal{N}$. Let us assume that there is a linear bounded operator $R: Z \to U$ such that the “nonlinearity” $\varphi(z) = Rz$ belongs to $\mathcal{N}$.

**Theorem 3.1.** ([17]) Suppose that $\chi(\cdot)$ is the Laplace transform of $K$ and the operator function $(I - \chi(\lambda)R)^{-1}$ has poles in the right half-plane and the frequency-domain condition from [18] is satisfied. Then there exists a bounded linear self-adjoint operator

$$P: W_{-\rho}^{1,2}(0, +\infty; Z) \to W_{-\rho}^{1,2}(0, +\infty; Z) \quad \text{such that}$$

$$\mathcal{C} := \{h \in W_{-\rho}^{1,2}(0, +\infty; Z) \mid (Ph, h)_{W_{-\rho}^{1,2}(0, +\infty; Z)} < 0\}$$

is a quadratic cone $\mathcal{C} \neq \emptyset$ in $W_{-\rho}^{1,2}(0, +\infty; Z)$ with the following properties:

a) There exists a constant $\beta > 0$ such that for any $h \in \mathcal{C}$ and any $\phi \in \mathcal{N}$

$$\lim_{t \to \infty} e^{-\beta t} \int_0^t \| \phi(z(s, h, \phi), s) \|_U^2 ds = \infty. \quad (3.4)$$

b) Any solution $z(\cdot, h, \phi)$ of (3.3) which does not satisfy (3.4) has the property $\int_0^\infty \|\phi(z(s, h, \phi), s)\|_U^2 ds < \infty$ and, consequently,

$$\phi(z(\cdot, h, \phi), \cdot) \in L^2(0, \infty; U) \quad \text{and} \quad z(\cdot, h, \phi) \in L^2(0, \infty; Z). \quad (3.5)$$

In the next subsection we show an application of this theorem.

## 3.2 Hyperbolic PDE's with boundary control as realizations of Volterra equations

Let us investigate the question how to suppress vibrations in a fluid conveying tube via control on the boundary. We consider for this a system of equations which is described in [12, 21].

The motion of an incompressible fluid is given for $t > 0$ in the acoustic approximation by the hyperbolic PDE's

$$\frac{\partial v}{\partial t} = a_1 \frac{\partial w}{\partial x}, \quad \frac{\partial w}{\partial t} = a_2 \frac{\partial v}{\partial x}, \quad x \in (0, 1), \quad (3.6)$$

where $a_1$ and $a_2$ are positive parameters, $v$ denotes the relative velocity of the fluid and $w$ denotes the pressure. The boundary conditions are given for $t > 0$ by

$$w(t, 1) = 0, \quad \frac{1}{2}w(t, 0) - v(t, 0) = -u(t), \quad (3.7)$$

where $u(\cdot)$ is a function (“boundary control”) which describes the relative displacement of the piston of a servomotor. The equation of the turbine for $t > 0$ is

$$T_a \frac{dq}{dt} + q(t) = u(t) + \frac{3}{2} w(0, t). \quad (3.8)$$

Here $q$ denotes the relative angular speed of the turbine, $T_a$ is a positive parameter. The regulator is described by the equation

$$T_r^2 \frac{d^2 \zeta}{dt^2} + T_k \frac{d \zeta}{dt} + c_0 \zeta + k\phi(\zeta) + q(t) = 0, \quad (3.9)$$
where $\zeta$ represents the displacement of the clutch of the regulator and $T_r, T_k, c_0$ and $k$ are positive parameters. The friction term is given by a continuous function $\phi: \mathbb{R} \to \mathbb{R}$ defined through a parameter $\kappa > 0$ by

$$ \phi(z) = \begin{cases} 1 & \text{if } z \ge \kappa, \\ \frac{1}{\kappa}z & \text{if } z \in (-\kappa, \kappa), \\ -1 & \text{if } z \le -\kappa, \end{cases} \text{ and thus satisfying the property } \phi(z) \ge 0, \forall z \in \mathbb{R}. $$

$$ \text{The equation of the servomotor is} \qquad T_s \frac{du}{dt} = \eta(t), \qquad (3.10) $$

where $T_s$ is a positive parameter and $\eta$ denotes the displacement of the slide value. The last condition is for $t > 0$ and with a positive parameter $\beta$

$$ \eta(t) - \zeta(t) + \beta u(t) = 0. \qquad (3.11) $$

A direct computation shows that the transfer function of the linear part of (3.6) – (3.11) which connects the (formal) Laplace transforms of $-\phi$ and $\dot{\zeta}$ is given by

$$ \chi(\lambda) = k \frac{\lambda (T_a \lambda + 1) (T_s \lambda + \beta) (\sinh \lambda \tau + \alpha \cosh \lambda \tau)}{(T_r^2 \lambda^2 + T_k \lambda + c_0) Q(\lambda) + R(\lambda)}, \qquad (3.12) $$

$$ \text{where} \quad \alpha = 2\sqrt{\frac{a_1}{a_2}}, \quad \tau = 1/\sqrt{a_1 a_2}, \qquad (3.12.a) $$

$$ Q(\lambda) = (T_s \lambda + \beta) (T_a \lambda + 1)(\alpha \cosh \lambda \tau + \sinh \lambda \tau), \quad R(\lambda) = 2 \cosh \lambda \tau - 2 \sinh \lambda \tau. $$

Note that $\chi(\lambda)$ can be written with some $c > 0$ as

$$ \chi(\lambda) = \frac{k}{T_r^2 \lambda + c} + \chi_1(\lambda). \qquad (3.13) $$

The representation (3.13) shows that $\chi_1(\lambda)$ is analytic in some halfplane $\{\text{Re} \lambda > -\varepsilon\}$ with $\varepsilon > 0$. From this it follows that $\chi_1(\lambda)$ has the Laplace original $K_1(t)$ which is absolute continuous, satisfies the inequalities

$$ |K_1(t)| \le \text{const} e^{-\varepsilon_0 t} \qquad (3.14) $$

with some $\varepsilon_0 > 0$ and such that $K_1$ and $\dot{K}_1$ belong to $L^2(0, \infty; \mathbb{R})$. Since the first part of (3.13) has the Laplace original $\frac{k}{T_r^2} e^{-ct/T_r^2}$ the whole original of $\chi(\lambda)$ can be represented as $K(t) = K_1(t) + \frac{k}{T_r^2} e^{-ct/T_r}$.

It is shown in [21] that any solution component $z(t) := \dot{\zeta}(t)$ from (3.9) can be written as

$$ z(t) = h(t) + \int_{0}^{t} K(t - \tau) \phi(z(\tau)) d\tau, \qquad (3.15) $$

where again $h$ is absolute continuous, satisfies an inequality of type (3.14) and $h, \dot{h}$ belong to $L^2(0, \infty; \mathbb{R})$. The quadratic constraints can be described in $Z = U = \mathbb{R}$ by the inequality

$$ \phi(z) (z - \kappa \phi(z)) \ge 0, \quad \forall z \in \mathbb{R}. \qquad (3.16) $$

Using the transfer function (3.13) and the constraints (3.16) we can verify the frequency-domain condition from [18]. A direct computation shows (see [21]) that if

$$ T_k(\alpha^2 - 1)\beta^2 \le \left(\frac{55}{32} + \alpha^2\right) (\beta T_a + T_s) \qquad (3.17) $$

is satisfied, the condition

$$ \alpha(T_k \beta^2 - (\beta T_a + T_s)) \ge 3 \tau \beta. \qquad (3.18) $$
is necessary and sufficient for the frequency-domain condition. The stability and instability domains of the denominator of $\chi(\lambda)$ where investigated in [12] and characterized in the $(T_k, T_r^2)$-plane by domains $\Omega_{st}$ and $\Omega_{unst}$, respectively. It follows now that under the conditions (3.16) - (3.18) for parameters from $\Omega_{st}$ or $\Omega_{unst}$ the solutions of the integral equation (3.15) (and the solutions of the PDE problem (3.6) – (3.11)) have the properties described by Theorem 3.1.

Particularly one can derive frequency-domain conditions for the point-wise strong (in the sense of $L^p$) convergence of solutions of the above fluid control system to the stationary set (attractor) consisting of a continuum of functions ([21]).

# 4 Convergence to the stationary set in systems with hysteresis

## 4.1 Evolutionary variational inequalities

It is widely recognized that the Dirac delta function $\delta$ plays an important role in neural network theory, e.g., for the description of integrate and fire neurons. We now show one way of introducting $\delta$ into the framework of nonlinear systems with hysteresis nonlinearity, which can be used for constructing continuum-type memories. Some applications of hysteretic memories are demonstrated in [8, 23]. The multivalued properties of hysteresis operators are described by variational inequalities or differential inclusions ([22]) which are defined in some abstract setup.

Suppose that $Y_0$ is a real Hilbert space with $(\cdot, \cdot)_0$ and $\|\cdot\|_0$ as scalar product resp. norm. Suppose also that $A: \mathcal{D}(A) \subset Y_0$ is a closed (unbounded) densely defined linear operator. The Hilbert space $Y_1$ is defined as $\mathcal{D}(A)$ equipped with the scalar product

$$ (y, \eta)_1 := ((\beta I - A)y, (\beta I - A)\eta)_0, \quad y, \eta \in \mathcal{D}(A), $$

where $\beta \in \rho(A)$ ($\rho(A)$ is the resolvent set of $A$) is an arbitrary but fixed number the existence of which we assume.

The Hilbert space $Y_{-1}$ is by definition the completion of $Y_0$ with respect to the norm $\|z\|_{-1} = \|(\beta I - A)^{-1}z\|_0$. Thus we have the dense and continuous embedding $Y_1 \subset Y_0 \subset Y_{-1}$ which is called Hilbert space rigging structure. The duality pairing $(\cdot, \cdot)_{-1,1}$ on $Y_1 \times Y_{-1}$ is the unique extension by continuity of the functionals $(\cdot, y)_0$ with $y \in Y_1$ onto $Y_{-1}$.

If $-\infty \le T_1 < T_2 \le +\infty$ are arbitrary numbers, we define the norm for Bochner measurable functions in $L^2(T_1, T_2; Y_j)$, $j = 1, 0, -1$, through $\|y\|_{2,j} := (\int_{T_1}^{T_2} \|y(t)\|_j^2 dt)^{1/2}$. For an arbitrary interval $J$ in $\mathbb{R}$ denote by $W(J)$ the space of functions $y(\cdot) \in L^2_{\text{loc}}(J; Y_1)$ for which $\dot{y}(\cdot) \in L^2_{\text{loc}}(J; Y_{-1})$ equipped with the norm defined for any compact interval $[T_1, T_2]$ by $\|y(\cdot)\|_{W(T_1, T_2)} := (\|y(\cdot)\|_{2,1}^2 + \|\dot{y}(\cdot)\|_{2,-1}^2)^{1/2}$. Suppose that $U, Z$ are real Hilbert spaces with scalar products $(\cdot, \cdot)_U, (\cdot, \cdot)_Z$ and norms $\|\cdot\|_U, \|\cdot\|_Z$ and $A \in \mathcal{L}(Y_1, Y_{-1})$, $B \in \mathcal{L}(U, Y_{-1})$, $C \in \mathcal{L}(Y_0, U)$ are linear operators. Assume that $\phi: \mathcal{D}(\phi) \subset W^{1,2}(0, T; Z) \times U \to W^{1,2}(0, T; U)$ is a strongly continuous hysteresis operator with $\mathcal{E}: Z \to 2^U$, $\mathcal{D}(\phi) = \{(z, u_0) \in W^{1,2}(0, T; Z) \times U \mid u_0 \in \mathcal{E}(z(0))\}$ and $\psi: Y_1 \to \mathbb{R} \cup \{+\infty\}, \psi \ne +\infty$, is a convex, lower semi-continuous map. Consider the variational inequality
$$ (\dot{y}(t) - Ay(t) - Bu(t), \eta - y(t))_{-1,1} + \psi(\eta) - \psi(y(t)) \geq 0, \quad \forall \eta \in Y_1 \text{ a.e. } t \in (0, T) \quad (4.1) $$

$$ u(t) = \phi(z, u_0)(t), \quad z(t) = Cy(t), \quad y(0) = y_0 \in Y_0, \quad u_0 \in \mathcal{E}(z(0)). \quad (4.2) $$

A pair of functions $\{y, u\} \in \mathcal{W}(0, T) \cap (C(0, T); Y_0) \times L^2(0, T; U)$ which satisfies (4.1), (4.2), is called solution of (4.1), (4.2) on $(0, T)$ with initial conditions $y(0) = y_0$ and $u(0) = u_0$; $y(\cdot)$ is the state trajectory, $u(\cdot)$ is the control.

A solution $\{y, u\}$ of (4.1), (4.2) with $y(t) \equiv \hat{y} \in Y_1$ and $u(t) \equiv \hat{u} \in U$ is called stationary. The set of all stationary solutions of (4.1), (4.2) is the stationary set $S$.

Any stationary solution $\{\hat{y}, \hat{u}\}$ satisfies the stationary variational inequality

$$ (-A\hat{y} - B\hat{u}, \eta - \hat{y})_{-1,1} + \psi(\eta) - \psi(\hat{y}) \geq 0, \quad \forall \eta \in Y_1 \quad (4.3) $$

$$ \hat{u} = \phi(\hat{z}, \hat{u}), \quad \hat{z} = C\hat{y}, \quad \hat{u} \in \mathcal{E}(\hat{z}). \quad (4.4) $$

## 4.2 Convergence to the stationary set

Let us introduce the following assumptions using additional properties of a control system with hysteresis operator ([22]).

### (A0) Existence of solutions.

The inequality (4.1), (4.2) has for arbitrary $y_0 \in Y_0$ and $u_0 \in \mathcal{E}(Cy_0(0))$ at least one solution $\{y, u\}$. The stationary set $S$, given by (4.3), (4.4), is non-empty.

### (A1) Generalized Clausius - Duhem inequality.

$$ \exists F_1 \in \mathcal{L}(Z, U) \exists F_2 \in \mathcal{L}(U, U) \quad \forall T \geq 0 \quad \forall z \in W^{1,2}(0, T; Z) \quad \forall u_0 \in \mathcal{E}(z(0)) : \int_0^T (\dot{\phi}(z, u_0)(t), F_1 \dot{z}(t))_U - (\dot{\phi}(z, u_0)(t), F_2 \dot{\phi}(z, u_0)(t))_U dt \geq 0. $$

### (A2) Counter clockwise circulation.

$$ \exists \varkappa \in \{-1, 1\} \quad \exists G_1 \in \mathcal{L}(U, Z) \quad \forall T \geq 0 \quad \forall z \in W^{1,2}(0, T; Z) \quad \forall u_0 \in \mathcal{E}(z(0)) \\ \exists \gamma(z(0)) \geq 0 : \quad \varkappa \int_0^T (G_1 \phi(z, u_0)(\tau), \dot{z}(\tau))_Z d\tau \geq -\gamma(z(0)). $$

### (A3) Controllability of the linear part.

The pair $(A, B)$ is $L^2$-controllable, i.e. $\forall y_0 \in Y_0 \quad \exists u(\cdot) \in L^2(0, +\infty; U)$ such that $\dot{y} = Ay + Bu$, $y(0) = y_0$, is well-posed in the variational sense on $(0, +\infty)$.

### (A4) Exponential stability of the system operator.

Any solution of $\dot{y} = Ay$, $y(0) \in Y_0$, is exponentially decreasing for $t \to +\infty$.

### (A5) Regularity of the system operator.

The operator $A \in \mathcal{L}(Y_1, Y_{-1})$ is regular, i.e. $\forall T > 0, y_0 \in Y_1, w_T \in Y_1$ and $\forall f \in L^2(0, T; Y_{-1})$ the solutions of $\dot{y} = Ay + f(t)$, $y(0) = y_0$, and of $\dot{w} = -A^+w + f(t)$, $w(T) = w_T$, are strongly continuous in the norm of $Y_1$. $A^+$ is the adjoint operator. Define the quadratic form

$$ \begin{gathered} \mathcal{F}(\zeta, \vartheta; \tau) := (\vartheta, F_1 C w)_U - (\vartheta, F_2 \vartheta)_U - \tau (G_1 C w, u)_U, \\ \zeta = (w, u) \in Y_0 \times U, \quad \vartheta \in U (F_1, F_2 \text{ from (A1)}, \tau \in \mathbb{R} \text{ a parameter}). \end{gathered} $$
(A6) Frequency domain condition.

$$ \exists \tau \in \mathbb{R}, \quad \varkappa\tau \le 0, \quad \exists \delta > 0 \quad \mathcal{F}^c(\tilde{\zeta}, \tilde{\vartheta}; \tau) \le -\delta|\tilde{\zeta}|^2 \\ \forall \tilde{\zeta} \in U^c \quad \forall \omega \in \mathbb{R}, \quad \tilde{\zeta} = (i\omega(i\omega I - A)^{-1}B\tilde{u}, \tilde{u}), \quad \tilde{\vartheta} = i\omega\tilde{u}. $$

Here $U^c$ is the complexification of $U$ and $\mathcal{F}^c$ is the Hermitian extension of $\mathcal{F}$.

(A7) Convexity and lower semi-continuity of the transformed hysteresis operator.
For any operator $P = P^* \in \mathcal{L}(Y_0, Y_1)$ such that

$$ P \in \mathcal{L}(Y_{-1}, Y_0) \cap \mathcal{L}(Y_0, Y_1) \quad \text{we have} \quad \begin{aligned} \psi(y_1) - \psi(y_1 - P(y_1 - y_2)) \\ + \psi(y_2) - \psi(y_2 + P(y_1 - y_2)) \ge 0, \quad \forall y_1, y_2 \in Y_1. \end{aligned} $$

On $Y_1$ the function $\psi_\rho(y) := \psi(y - Py) - \psi(y)$ is convex and lower semi-continuous.

**Theorem 4.1.** ([18]) Under the above assumptions any solution $\{y, u\}$ of (4.1), (4.2) converges strongly in $Y_0 \times U$ to the stationary set $S$ as $t \to \infty$.

**Example 4.1.** We consider the boundary problem for the temperature $\theta(x,t)$ on the string $(0,1)$ with hysteresis control on the boundary given by

$$ \begin{aligned} \theta_t &= \theta_{xx} - b\theta, \quad x \in (0, 1), \quad \theta(x, 0) = \theta_0(x), \quad \theta_x(0, t) = 0, \\ \theta_x(1, t) &= \rho \phi(z, u_0)(t), \quad z(t) = \int_0^1 \theta(x, t) dx. \end{aligned} \quad (4.5) $$

Here $z(t)$ is the mean temperature at $t$, $b > 0$ and $\rho \in \mathbb{R} \setminus \{0\}$ are parameters, $\theta_0(x)$ is an initial function, $\phi : \mathcal{D}(\phi) \subset W^{1,1}(0,T) \times \mathbb{R} \to W^{1,1}(0,T)$, $(z, u_0) \in \mathcal{D}(\phi) \mapsto \phi(z, u_0)(\cdot) \in \mathcal{E} : \mathbb{R} \to 2^\mathbb{R}$ s. th. $\mathcal{D}(\phi) = \{(z, u_0) \in W^{1,1}(0,T) \times \mathbb{R} \mid u_0 \in \mathcal{E}(z(0))\}$ is a continuous hysteresis operator.

**Quadratic constraints:**

(A1) : $0 \le \dot{\phi}(z, u_0)(t)\dot{z}(t) \le (\dot{z}(t))^2 \quad \forall z \in W^{1,1}(0, T)$

(A2) : $\varkappa = -1$, $\varkappa \int_{t_1}^{t_2} \phi(z, u_0)(t) \dot{z}(t) dt \geq -\gamma(z(0)) \forall z \in W^{1,1}(0, T)$, $\gamma(z(0)) \geq 0$

**Interpretation as variational inequality:**

$$ Y_0 := L^2(0,1), \quad (v,w)_0 := \int_0^1 v(x)w(x)dx, \quad \forall v,w \in L^2(0,1). $$

$$ Y_1 := W^{1,2}(0,1) \quad (v,w)_1 := \int_0^1 (v(x)w(x) + v'(x)w'(x))dx, \quad \forall v,w \in W^{1,2}(0,1) $$

$$ \begin{aligned} Y_{-1} &:= Y_1' \text{ dual. } A \in \mathcal{L}(Y_1, Y_{-1}) \text{ is given by } (Av, w)_{-1,1} := \int_0^1 (Av)(x)w(x)dx \\ &= -\int_0^1 (v'(x)w'(x) + bv(x)w(x))dx, \quad \forall v,w \in W^{1,2}(0,1). \end{aligned} $$

$$ U := \mathbb{R}, \ B \in \mathcal{L}(\mathbb{R}, Y_{-1}) \text{ is defined by} $$

$$ \begin{gathered} (Bu, w(x))_{-1,1} = \rho uw(1), \quad \forall v \in \mathbb{R}, \forall v \in W^{1,2}(0,1), \\ \text{i.e. } B = [\rho \delta(x-1)], \delta \text{ is the Dirac's distribution.} \end{gathered} $$

$$ Z := \mathbb{R}, \ C \in \mathcal{L}(Y_0, Z) \text{ is given by } C_v := \int_0^1 v(x)dx, \quad \forall v \in L^2(0,1). $$

It follows that assumptions (A0) – (A5) are satisfied. The verification of (A6) and (A7) is shown in [17]. It follows from Theorem 4.1 that any solution of (4.5) converges to the stationary set for $t \to \infty$, see Fig. 4.1 and 4.2 for the case of a play operator.
<figure><img src="image_4.png" /><figcaption>Fig. 4.1 Intersection of $u = \frac{b}{\rho}z$ with the graph of the hysteresis</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4.2 Continuum of stationary temperature fields</figcaption></figure>

Note that parabolic equations with Dirac's delta function at the right-hand side arise also as Fokker-Planck-Kolmogorov equation for the probability density of neurons in nonlinear noisy leaky integrate and fire models of neural networks [5]. Convergence to the stationary set is connected with the construction of global attractors as working memory [7, 9, 20]. General Hausdorff dimension and stability properties of such attractors for smooth and non-smooth dynamical systems and cocycles (non-autonomous dynamical systems) are considered in [10].

## 5 Conclusion

With this paper, we could present an approach to bring continuum-type memories and neural networks together. The convergence behavior of continuum-type memories in the representation of a trajectory from input data set to an output class can be used to improve the understanding and further analysis for neural networks. For distance measurement methods, which are mainly represented by temporally offset pulses that can be better recognized by specific neural networks (e.g., spiking neural networks), this is a subject for further research to be verified in experiments.

Part 3 and 4 of the work are supported by the 2020-2021 program Leading Scientific Schools of the Russian Federation (project NSh-2624.2020.1) and Saint Petersburg State University (ID 75206671).

## References

1. Aggarwal, C.C., Reddy, C.K.: Data Clustering - Algorithms and Applications. CRC Press, Boca Raton, Fla (2013)

2. AI and Robotics for GeoEnvironmental Modeling and Monitoring (AIRGEMM), https://tufreiberg.de/airgemm (accessed on 12 March 2021)

3. Kisner, H., Thomas, U.: Segmentation of 3d point clouds using a new spectral clustering algorithm without a-priori knowledge. In: Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, pp. 315-322. INSTICC, SciTePress (2018). https://doi.org/10.5220/0006549303150322
4. Nakagawa, M.: Point Cloud Clustering Using Panoramic Layered Range Image. IntechOpen (2018). https://doi.org/10.5772/intechopen.76407

5. Carrillo, J., Perthame, B., Salort, D., Smets, D.: Qualitative properties of solutions for the noisy integrate & fire model in computational neuroscience. Computational Neuroscience, Nonlinearity **28** (9), 3365–3388 (2015)

6. Comsa, I. M., Pottempa, K., Versari, L., Fischbacher, T., Gesmundo, A., Alakuijala, J.: Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation. (2020), https://arxiv.org/pdf/1907.13223.pdf

7. Eliasmith, C.: A unified approach to building and controlling spiking attractor networks. Neural Computation **17** (6), 1276–1314 (2005)

8. Farrokh, M., Dizaji, M., Dizaji, F., Moradinasab, N.: Universal hysteresis identification using extended Preisach Neural Network. (2019), https://arxiv.org/pdf/2001.01559.pdf

9. Fusi, S.: Hebbian spike-driven synaptic plasticity for learning patterns of mean firing rates. Biological Cybernetics **87** (5-6), 459–470 (2002)

10. Kuznetsov, N., Reitmann, V.: Attractor Dimension Estimates for Dynamical Systems: Theory and Computation. Springer International Publishing (2021)

11. Mutto, C.D.; Zanuttigh, P.; Cortelazzo, G.M. *Time-of-Flight Cameras and Microsoft Kinect™;* Springer US: Boston, MA, (2012)

12. Neimark, Yu. I.: On Lyapunov stability of systems with distributed wave units. Uchenye Zapiski Gor'kovskogo Gos. Universiteta, Ser. Fiz., XVI (1950) (Russian)

13. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. abs/1612.00593 (2016), http://arxiv.org/abs/1612.00593

14. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. abs/1706.02413 (2017), http://arxiv.org/abs/1706.02413

15. Rajpura, P.S., Goyal, M., Bojinov, H., Hegde, R.S.: Dataset augmentation with synthetic images improves semantic segmentation. CoRR abs/1709.00849 (2017), http://arxiv.org/abs/1709.00849

16. Reitmann, S.; Neumann, L.; Jung, B. BLAINDER—A Blender AI Add-On for Generation of Semantically Labeled Depth-Sensing Data. Sensors 2021, 21, 2144. https://doi.org/10.3390/s21062144

17. Reitmann, V.: Convergence in evolutionary variational inequalities with hysteresis nonlinearities. In: Proceedings of Equadiff 11, Bratislava, pp. 395–404 (2005)

18. Reitmann, V.: Realization theory methods for the stability investigation of nonlinear infinite-dimensional input-output systems. MATHEMATICA BOHEMICA, **136** (2), 185–194 (2011)

19. Reitmann, V., Zyryanov, D.: The global attractor of a multivalued dynamical system generated by a two-phase heating problem. Differential Equations and Control Processes **4**, 118–138 (2017) (Russian)

20. Seeholzer, A., Deger, M., Gerstner, W.: Stability of working memory in continuous attractor networks under the control of short-term plasticity. PLoS Comput. Biol. **15** (4) (2019)

21. Smirnova, V. B.: On the asymptotic behavior of a class of control systems with distributed parameters. Avtomatika i Telemekhanika **10**, 5–12 (1973) (Russian)

22. Visintin, A.: Differential Models of Hysteresis. Springer-Verlag Berlin Heidelberg (1994)

23. Wei, Jyh-Da, Sun, Chuen-Tsai: Constructing hysteretic memory in neural networks. IEEE Trans. Syst. Man Cybern. Part B **30** (4), 601–609 (2000) Journal **2** (5), 99–110 (2016)

24. Yi, L., Kim, V.G., Ceylan, D., Shen, I.C., Yan, M., Su, H., Lu, C., Huang, Q., Sheffer, A., Guibas, L.: A scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph. **35**(6) (2016). https://doi.org/10.1145/2980179.2980238, https://doi.org/10.1145/2980179.2980238