abstract: We consider the problem of optimally executing an order involving multiple crypto-assets, sometimes called tokens, on a network of multiple constant function market makers (CFMMs). When we ignore the fixed cost associated with executing an order on a CFMM, this optimal routing problem can be cast as a convex optimization problem, which is computationally tractable. When we include the fixed costs, the optimal routing problem is a mixed-integer convex problem, which can be solved using (sometimes slow) global optimization methods, or approximately solved using various heuristics based on convex optimization. The optimal routing problem includes as a special case the problem of identifying an arbitrage present in a network of CFMMs, or certifying that none exists.
author: Guillermo Angeris
`email@example.com`; Tarun Chitra
`firstname.lastname@example.com`; Alex Evans
`email@example.com`; Stephen Boyd
`firstname.lastname@example.com`
bibliography: citations.bib
date: December 2021
title: Optimal Routing for Constant Function Market Makers

# Introduction

Decentralized exchanges (DEXs) are a popular application of public blockchains that allow users to trade assets without the need for a trusted intermediary to facilitate the exchange. DEXs are typically implemented as *constant function market makers* (CFMMs) . In CFMMs, liquidity providers contribute reserves of assets. Users can then trade against these reserves by tendering baskets of assets in exchange for other baskets. CFMMs use a simple rule for accepting trades: a trade is only valid if the value of a given function at the post-trade reserves (with a small adjustment to account for fees collected) is equal to the value at the pre-trade reserves. This function is called the *trading function* and gives CFMMs their name. A common example of a trading function is the constant product popularized by Uniswap , wherein a trade is only accepted if it preserves the product of the reserve amounts.

CFMMs have quickly become one of the most popular applications of public blockchains, facilitating several billion dollars of trading volume per day. As DEXs have grown in popularity, so have the number of CFMMs and assets offered, creating complexity for traders who simply want to maximize their utility for trading one basket of assets for another. As a result, several "DEX aggregators" have emerged to route orders across multiple CFMMs on behalf of users. These aggregators currently execute several billion dollars per month across all DEXs on Ethereum . At the same time, CFMM platforms such as Uniswap offer software for routing orders across the subset of CFMMs they support .

While the properties of individual CFMMs have been studied extensively (see, , ) it is more common for users to want to access liquidity on multiple CFMMs to minimize their total trading costs. In this paper, we study the optimal routing problem for CFMMs. We consider a user who can trade with multiple CFMMs in order to exchange one basket of assets for another and ask how one should perform such trades optimally. We show that, in the absence of fixed costs, the optimal routing problem can be formulated as a convex optimization problem, which is efficiently solvable. As an (important) sub-case, we demonstrate how to use this problem to identify arbitrage opportunies on a set of CFMMs. Our framework encompasses the routing problems considered in prior work  and offers solutions in the more general case where users seek to trade any basket of assets for any other basket across any set of CFMMs whose trading functions are concave and not necessarily differentiable. When including transaction costs, we show that the optimal routing problem is a mixed-integer convex problem, which permits (potentially computationally intensive) global solutions as well as approximate solutions using heuristics based on convex optimization.

#### Outline.

We describe the optimal routing problem in §, ignoring the fixed transaction costs. In § we give the optimality conditions for the optimal routing problem, and give conditions under which the optimal action is to not trade at all. We give some examples of the optimal routing problem in §, including as a special case the detection of arbitrage in the network. We give a simple numerical example in §. In § we show how to add fixed transaction costs to the optimal routing problem, and briefly describe some exact and approximate solution approaches.

# Optimal routing problem

#### Network of CFMMs.

We consider a set of $m$ CFMMs, denoted $i=1, \ldots, m$, each of which trades multiple tokens from among a universe of $n$ tokens, labeled $j=1 \ldots, n$. We let $n_i$ denote the number of tokens that CFMM $i$ trades, with $2 \leq n_i \leq n$. We can think of the $m$ CFMMs as the vertices of a network or hypergraph, with $n$ hyper-edges, each corresponding to one of the $n$ assets, adjacent to those CFMMs that trade it. Alternatively we can represent this as a bipartite graph, with one group of $m$ vertices the CFMMs, and the other group of $n$ vertices the tokens, with an edge between a CFMM and a token if the CFMM trades the token.

A simple example is illustrated as a bipartite graph in figure . In this network there are $m=5$ CFMMs, which trade subsets of $n=3$ tokens. CFMM 1 trades all three tokens, so $n_1=3$; the remaining 4 CFMMs each trade pairs of tokens, so $n_2= \cdots = n_5=2$.

#### Global and local token indices.

We use multiple indices to label the tokens. The *global* index uses the token labels $1, \ldots, n$. CFMM $i$, which trades a subset of $n_i$ of the $n$ tokens, has its *local* token index, $j=1, \ldots, n_i$. To link the global and local indexes, we introduce matrices $A_i \in
\reals^{n \times n_i}$, $i=1\ldots, m$, where $(A_i)_{jk} = 1$ if token $k$ in the CFMM $i$'s local index corresponds to global token index $j$, while $(A_i)_{jk} =0$ otherwise.

As an example, consider the simple network shown in figure . CFMM 1 trades all three tokens, with its local indexing identical to the global indexing, so $A_1$ is the $3\times 3$ identity matrix. As a more interesting example, consider CFMM 4, which trades two tokens, labeled 1 and 2 in the local indexing, but 1 and 3 in the global indexing, so $$A_4 = \begin{bmatrix}
    1 & 0\\
    0 & 0\\
    0 & 1\\
 \end{bmatrix}.$$

#### CFMM tendered and received baskets.

For CFMM $i$ we denote the tendered and received baskets as $\Delta_i,~\Lambda_i \in \reals_+^{n_i}$. These are quantities of tokens (in the local indexing) that we give to, and receive from, CFMM $i$ in a proposed trade.

#### CFMM trading semantics.

The proposed trade $(\Delta_i,\Lambda_i)$ is valid and accepted by CFMM $i$ provided $$\varphi_i(R_i+\gamma \Delta_i - \Lambda_i) = \varphi_i(R_i),$$ where $\varphi: \reals^{n_i}_+ \to \reals$ is the trading function, $R_i \in \reals_+^{n_i}$ are the current reserves, and $\gamma_i \in (0,1]$ is the trading fee for CFMM $i$. We will assume that the trading functions $\varphi_i$ are concave and increasing functions. (For much more detail, see .)

Examples of trading functions are the sum function, with $$\varphi_i(R)= R_1+ \cdots + R_{n_i},$$ the product or geometric mean function $$\varphi_i(R)= (R_1 \cdots R_{n_i})^{1/n_i},$$ and a generalization, the weighted geometric mean function $$\varphi_i(R)= R_1^{w_1} \cdots R_{n_i}^{w_{n_i}},$$ with weights $w>0$, $\ones^Tw=1$, where $\ones$ is the vector with all components one.

#### Network trade vector.

The CFMM tendered and received baskets $\Delta_i$ and $\Lambda_i$ can be mapped to the global indices as the $n$-vectors $A_i \Delta_i$ and $A_i \Lambda_i$, which give the numbers of tokens tendered to and received from CFMM $i$, using the global indices for the tokens. Summing the difference of received and tendered tokens over all CFMMs we obtain the *network trade vector* $$\Psi = \sum_{i=1}^m A_i (\Lambda_i - \Delta_i).$$ This is an $n$-vector, which gives the total net number of tokens tendered to the CFMMs in the network. We interpret $\Psi$ as the net network trade vector. If $\Psi_k \geq 0$, it means that we do not need to tender any of token $k$ to the network. This does not mean we that do not trade token $k$; it only means that in our trading with the CFMMs, we receive at least as much as token $k$ as we tender.

#### Network trade utility.

We introduce a utility function $U:\reals^n \to \reals \cup \{-\infty\}$ that gives the utility of a trade $\Psi$ to a trader as $U(\Psi)$. We will assume that $U$ is concave and increasing. Infinite values of $U$ are used to impose constraints; we consider a proposed trade with $U(\Psi) = -\infty$ as unacceptable. As an important example, consider the constraint $\Psi + h \geq 0$, where $h \in \reals_+^n$ is the vector of a user's current holdings of tokens. This constraint specifies that the post-trade holdings $\Psi+ h$ should be nonnegative, , we cannot enter into a trade that requires more of any token than we currently have on hand. To express this in the utility function, we define $U(z)$ to be $-\infty$ when $z+g \not\geq 0$. (This modified utility is also concave and increasing.)

There are many possible choices for $U$. Perhaps the simplest is the linear utility function $U(z) = \pi^Tz$, with $\pi \in \reals_{++}$, where we interpret $\pi_i$ as the trader's internal or private value or price of token $i$. Several other choices are discussed in .

#### Optimal routing problem.

We wish to find a set of valid trades that maximizes the trader's utility. This optimal routing problem can be expressed as

The variables here are $\Psi \in \reals^n$, $\Lambda_i \in \reals_+^{n_i}$, $\Delta_i \in \reals^{n_i}_+$, $i=1, \ldots, m$. The data are the utility function $U$, the global-local matrices $A_i$, and those associated with the CFMMs, the trading functions $\varphi_i$, the trading fees $\gamma_i$, and the reserve amounts $R_i \in \reals^{n_i}_+$.

#### Convex optimal routing problem.

Unless the trading functions are affine, the optimal routing problem is not a convex optimization problem. We can, however, form an equivalent problem that is convex. To do this we replace the equality constraints with inequality constraints:

This problem is evidently convex  and can be readily solved.

We will show that any solution of is also a solution of . This the same as showing that for any solution of , the inequality constraints hold with equality. Suppose that $\Delta_i^\star$ and $\Lambda_i^\star$ are feasible for , but $\varphi_k(R_k + \gamma_k \Delta^\star_k - \Lambda^\star_k) > \varphi_k(R_k)$ for some $k$. This means we can find $\tilde \Lambda > \Lambda^\star_k$ which satisfies $\varphi_k(R_k + \gamma_k \Delta^\star_k - \tilde \Lambda) \geq \varphi_k(R_k)$. The associated trade vector $\tilde \Psi = \Psi^\star + A_k \Lambda_k$ satisfies $\tilde \Psi \geq \Psi^\star$, $\tilde \Psi \neq \Psi^\star$, , at least one component of $\tilde \Psi$ is larger that the corresponding component of $\Psi^\star$. It follows that $U(\tilde \Psi) > U(\Psi^\star)$, so $\Psi^\star$ is not optimal and therefore cannot be a solution. We conclude that any solution of satisfies the inequality constraints as equality, and so is optimal for .

A similar statement holds in the case when $U$ is only nondecreasing, and not (strictly) increasing. In this case we can say that there is a solution of that is optimal for . One simple method to find such a solution is to solve with objective $U(\Psi)+\epsilon \ones^T\Psi$, where $\epsilon$ is small and positive. The objective for this problem is increasing. In principle we can let $\epsilon$ go to zero to recover a solution of the original problem; in practice choosing a single small value of $\epsilon$ works.

#### Consequences of convexity.

Since the problem is convex, it can be reliably and quickly solved, even for large problem instances . Domain specific languages for convex optimiztion such as CVXPY  or JuMP can be used to specify the optimal routing problem in just a few lines of code; solvers such as ECOS , SCS , or Mosek  can be used to solve the problem.

#### Implementation.

The solution to  provides the optimal values $(\Delta_i,\Lambda_i)$ that one must trade with each CFMM $i$. Note that we do not explicitly consider the ideal execution of these trades, as these will depend on the semantics of the underlying blockchain that the user is interacting with. For example, users may wish to execute trades in a particular sequence, starting with an initial portfolio of assets and updating their asset composition after each transaction until the final basket is obtained. Users may alternatively use flashloans  to atomically perform all trades in a single transaction, netting out all trades before repaying the loan at the end of the transaction.

# Optimality conditions

Assuming that $U$ is differentiable (which it need not be), the optimality conditions of problem  are feasibility, and the dual conditions U() = , and \_i\_i \_i(R_i + \_i\_i - \_i) A_i\^T \_i\_i(R_i + \_i \_i - \_i), i=1, …, m, where $\nu \in \reals^n$, $\lambda_i \in \reals_+$, $i=1,\ldots, m$ are the Lagrange multipliers. (We derive these conditions in appendix .)

The first condition has a very simple interpretation: $\nu$ is the vector of marginal utilities of the tokens. The second set of conditions  has a simple interpretation that is very similar to the one given in  for a single CFMM. The term $\nabla \varphi_i(R_i + \gamma_i\Delta_i - \Lambda_i)$, which we will write as $P_i \in \reals^{n_i}_+$, can be interpreted as the unscaled prices of the tokens that CFMM $i$ trades , in the local indexing. Plugging into ,we get \_i\_iP_i A_i\^TU() \_iP_i, i=1, …, m. The middle term is the vector of marginal utilities of the tokens that CFMM $i$ trades, in the local indexing. These marginal utilities must lie between a multiple of the discounted prices, scaled, and the same prices, adjusted by $\gamma_i$.

We can also recognize the conditions  as those for the problem of finding an optimal trade for CFMM $i$ alone, with the linear utility $U_i(z) = \pi_i^T z$, where $\pi_i = A_i^T \nabla U(\Psi)$. (See .) This is very appealing: it states that the optimal trades for the network of CFMMs are also optimal for each CFMM separately, when they use a linear utility, with prices equal to the marginal utility of the overall trade.

#### Non-differentiable utility.

When $U$ is not differentiable, the optimality conditions are the same, but in we substitute a supergradient $g \in -\partial (-U)(\Psi)$ for the gradient $\nabla U(\Psi)$, where $\partial$ denotes the subdifferential. When $U$ is not differentiable at $\Psi$, there are multiple such $g$'s, which we can consider to be multiple marginal utilities. The optimality condition is that hold, with $\nabla U(\Psi)$ replaced with $g$, for any $g \in -\partial (-U)(\Psi)$.

#### No-trade condition.

From the optimality conditions we can derive conditions under which the optimal trades are zero, , we should not trade. We will assume that $U(0) > -\infty$, , $\Psi=0$ is feasible; if this is not the case, then evidently $\Psi=0$ is not optimal. The zero trade $$\Psi=0, \qquad \Delta_i = \Lambda_i = 0, \quad i=1, \ldots, m,$$ is feasible for , so the optimality condition is \_i\_i P_i A_i\^TU(0) \_iP_i, i=1, …, m, where $P_i = \nabla \varphi_i(R_i)$ is the unscaled price of CFMM $i$ at the current reserves $R_i$. This condition is a generalization of the no-trade condition given in  for one CFMM, to the case where there are multiple CFMMs. When $U$ is not differentiable, we replace $\nabla U(0)$ with a supergradient $g \in -\partial (-U)(0)$.

# Examples

#### Linear utility.

Consider the linear utility $U(z)=\pi^Tz$, with $\pi >0$. In this case the optimal routing problem is separable across the CFMMs, since the objective is a sum of functions of $\Lambda_i-\Delta_i$, and constraints are also only on $(\Lambda_i,\Delta_i)$. It follows that we can solve the optimal routing problem by solving $m$ single CFMM problems independently, using linear utilities with prices given by $\pi$.

#### Liquidating a basket of tokens.

Suppose we start with an initial holding (basket) of tokens $h^\text{init} \in \reals_+^n$ and wish to convert them all to token $k$. We use the utility function $$U(z) = \left\{ \begin{array}{ll} z_k & h^\text{init} + z \geq 0\\
-\infty & \mbox{otherwise.} \end{array}\right.$$ (This utility is nondecreasing but not increasing.)

A special case of this problem is converting one token into another, , when $$h^\mathrm{init} = t e_j,$$ where $t \ge 0$ and $e_j \in \reals^n$ is the $j$th unit vector. In this special case, we can write the optimal value of the optimization problem, which we will call $u(t)$, as a function of $t$. It is not hard to show that $u$ is nonnegative and increasing. This function is also concave as it is the partial maximization of a concave function (over all variables except $t$). We show an example instance of this problem, along with an associated function $u$, in §.

#### Purchasing a basket of tokens.

This is the opposite of liquidating a basket of tokens. Here too we start with initial token holdings $h^\text{init} \in \reals_+^n$, and end up with the holdings $h^\text{init} +  \Psi$. Let $h^\text{des} \in \reals_+^n$ be our target basket; we wish to end up with the largest possible multiple of this basket. Let $\mathcal K \subseteq \{1, \ldots, n\}$ denote the set of indices for which $h^\text{des}_i>0$, , the indexes associated with tokens in the desired basket. We seek to maximize the value of $\alpha$ for which $h^\text{init} + \psi \geq \alpha h^\text{des}$. To do this we use the utility function $$U(z) = \left\{ \begin{array}{ll} \min_{i \in \mathcal K} (h_i^\text{init}+\Psi_i)/
h_i^\text{des} & h^\text{init} + z \geq 0\\
-\infty & \mbox{otherwise.} \end{array}\right.$$

#### Arbitrage detection.

An arbitrage is a collection of valid CFMM trades with $\Psi \geq 0$ and $\Psi \neq 0$, , a set of trades for the CFMMs in which we tender no tokens, but receive a positive amount of at least one token. The optimal routing problem can be used to find an arbitrage, or certify that no arbitrage exists.

Consider any $U$ that is increasing, with domain $\{\Psi \mid U(\Psi)>-\infty \} = \reals_+^n$. Evidently there is an arbitrage if and only if there is a nonzero solution of the routing problem, which is the same as $U(\Psi^\star) > U(0)$, where $\Psi^\star$ is optimal. So by solving this optimal routing problem, we can find an arbitrage, if one exists.

#### No-arbitrage condition.

Using the version of for nondifferentiable $U$ we can derive conditions under which there is no arbitrage. We consider the specific utility function $$U(\Psi) = \left\{ \begin{array}{ll} \ones^T \Psi & \Psi \geq 0\\
-\infty & \mbox{otherwise},
\end{array}\right.$$ where $\ones$ denotes the vector with all entries one. This utility is the total number of tokens received, when they are nonnegative. Its supergradient at $0$ is $$-\partial (-U)(0) = [1,\infty)^n = \{ g\mid g \geq \ones \}.$$ The condition becomes: There exists $g \geq \ones$, and $\lambda_i \geq 0$, for which $$\gamma_i\lambda_i P_i \le A_i^T g \le \lambda_iP_i, \quad i=1, \ldots, m.$$ By absorbing a scaling of $g$ into the $\lambda_i$, we can say that $g>0$ is enough.

This makes sense: it states that there is no arbitrage if we can assign a set of positive prices (given by $g$) to the tokens, for which no CFMM would trade. In the (unrealistic) case when $\gamma_i=1$, , there is no trading cost, the no arbitrage condition is that there exists a global set of prices for the tokens, $g$, consistent with the local prices of tokens given by $\lambda_iP_i$.

# Numerical example

The Python code for the numerical example we present here is available at

`https://github.com/angeris/cfmm-routing-code`.

The optimization problems are formulated and solved using CVXPY . A listing of the core of the code is given in appendix .

#### Network.

We consider the network of 5 CFMMs and 3 tokens shown in figure . The trading functions, Fee parameters, and reserves are listed in table .

| **CFMM** | **Trading function $\varphi_i$** | **Fee parameter $\gamma_i$** | **Reserves** $R_i$ |
|:---|:---|:---|:---|
| 1 | Geometric mean, $w=(3, 2, 1)$ | 0.98 | (3, .2, 1) |
| 2 | Product | 0.99 | (10, 1) |
| 3 | Product | 0.96 | (1, 10) |
| 4 | Product | 0.97 | (20, 50) |
| 5 | Sum | 0.99 | (10, 10) |

CFMM attributes.

#### Problem and utility.

We wish to trade an amount $t\geq 0$ of token 1 for the maximum possible amount of token 3. This is a special case of the problem of liquidating a basket of tokens, as described in §, with initial holdings $h^\text{init}=t e_1$. The utility function is $U(z) = z_3$, provided $z+h^\text{init} \geq 0$, and $U(z) = -\infty$ otherwise. We let $u(t)$ denote the maximum amount of token 3 we can obtain from the network when we tender token 1 in the amount $t$.

#### Results.

We solve the optimal routing problem for many values of $t$, from $t=0$ to $t=50$. The amount of token $3$ we obtain in shown in figure . We see that $u(0) > 0$, which means there is an arbitrage in this network; indeed, there is a set of trades that requires giving zero net tokens to the network, but we receive an amount around 7 of token 3.

The associated optimal trades are shown in figure . We can see many interesting phenomena here. At $t=0$ we see the arbitrage trades, indeed, the arbitrage trades that yield the largest amount of token 3. Several asset flows reverse sign as $t$ varies. For example, for $t<11$, we receive token 1 from CFMM 1, whereas for $t>11$, we tender token 1 to CFMM 1. We can also see that the sparsity pattern of the optimal trades changes with $t$.

We illustrate the changing signs and sparsity of the optimal trades in figure , which shows the optimal trades for $t=0$, $t=20$, and $t=50$. We plot whether each token is tendered to or received from each CFMM using color coded edges. A red edge connecting a token to a CFMM means that the CFMM is receiving this token, while a blue edge denotes that the CFMM is tendering this token. A dashed edge denotes that the CFMM neither tenders nor receives this token.

Even in this very simple example, the optimal trades are not obvious and involve trading with and among multiple CFMMs. For larger networks, the optimal trades are even less obvious.

# Fixed transaction costs

Our optimal routing problem includes the trading costs built into CFMMs, via the parameters $\gamma_i$. But it does not include the small fixed cost associated with any trade. In this section we explore how these fixed transaction costs can be incorporated into the optimal routing problem.

We let $q_i \in \reals_+$ denote the fixed cost of executing a trade with CFMM $i$, denominated in some numeraire. We pay this whenever we trade, , $\Lambda_i - \Delta_i \neq 0$. We introduce a new set of Boolean variables into the problem, $\eta \in \{0, 1\}^m$, with $\eta_i = 1$ if a nonzero trade is made with CFMM $i$, and $\eta_i = 0$ otherwise, so the total fixed transaction cost is $q^T\eta$. We assume that there is a known maximum size of a tendered basket with CFMM $i$, which we will denote $\Delta^\mathrm{max}_i \in \reals_+^{n_i}$. We can then express the problem of maximizing the utility minus the fixed transaction cost as $$\label{eq:micp}
\begin{aligned}
    & \text{maximize} &&  U(\Psi) - q^T\eta\\
    & \text{subject to} && {\textstyle \Psi =\sum_{i=1}^m A_i(\Lambda_i - \Delta_i)}\\
    &&& \varphi_i(R_i + \gamma_i \Delta_i - \Lambda_i) \ge \varphi_i(R_i), \quad i=1, \dots, m\\
    &&& 0 \le \Delta_i \le \eta_i\Delta^\mathrm{max}_i, \quad \Lambda_i \ge 0, \quad i=1, \dots, m,\\
    &&& \eta \in\{0, 1\}^m,
\end{aligned}$$ where the variables are $\Psi$, $\Lambda_i$, $\Delta_i$, $i=1, \ldots, m$, and $\eta \in \{0,1\}^m$.

The optimal routing problem with fixed costs is a mixed-integer convex program (MICP). It can be solved exactly, possibly with great computational effort, using global optimization methods, with MICP solvers such as Mosek  or Gurobi . When $m$ is small (say, under ten or so), it can be practical to solve it by brute force, by solving the convex problem we obtain for each of the $2^m$ feasible values of $\eta$.

#### Approximate solution methods.

Many approximate methods have the speed of convex optimization, and (often) produce good approximate solutions. For example we can solve the relaxation of obtained by replacing the constraints on $\eta$ to $\eta \in[0,1]^m$ (which gives a convex optimization problem). After that we set a threshold $t\in (0,1)$ for the relaxed optimal values $\eta^\text{rel}$, and take $\eta_i = 1$ when $\eta^\text{rel}\geq t$ and and $\eta_i = 0$ when $\eta^\text{rel}<t$. We fix these values of $\eta$ and then solve the resulting convex problem. This could be done for a modest number of values of $t$; we take the solution found with the largest objective (including the fixed costs $q^T\eta$).

An alternative is a simple randomized method. We interpret $\eta^\text{rel}_i$ as probabilities, and generate $\eta \in
\{0,1\}$ randomly using these probabilities. We then solve the convex problem associated with this choice of $\eta$. We can repeat this procedure a modest number of times, and pick the feasible point with the highest payoff.

# Derivation of optimality conditions

To derive condition , we introduce the Lagrangian of the convex optimal routing problem , $$\begin{gathered}
\mathcal{L}(\Psi, \Delta, \Lambda, \nu, \lambda, \eta, \xi) =
-U(\Psi) + \nu^T\left(\Psi - \sum_{i=1}^m A_i(\Lambda_i - \Delta_i)\right) \\
+ \sum_{i=1}^m \lambda_i(\varphi_i(R_i) -
\varphi_i(R_i+\gamma_i\Delta_i - \Lambda_i)) - \sum_{i=1}^m
(\eta_i^T\Delta_i + \xi_i^T\Lambda_i).
\end{gathered}$$ Here, $\nu \in \reals^n$, $\lambda \in \reals^m_+$, and $\eta_i, \xi_i \in \reals^{n_i}_+$, $i=1, \dots, m$. These are the dual variables or Lagrange multipliers for the equality constraint, the relaxed inequality constraints, and nonnegativity constraints for the tendered and received baskets of assets for each CFMM, respectively.

The optimality conditions are (primal) feasibility, and (the dual conditions) $$\nabla_\Psi \mathcal{L} = 0, \quad \nabla_\Delta\mathcal{L}=0, \quad
\nabla_\Lambda\mathcal{L} = 0.$$ The first of these is $$-\nabla U(\Psi) + \nu = 0,$$ while the latter two are $$-A_i^T\nu + \lambda_i\nabla\varphi_i(R_i+\gamma_i\Delta_i - \Lambda_i) - \xi_i = 0,
\qquad A_i^T\nu - \gamma_i\lambda_i\nabla\varphi_i
(R_i+\gamma_i\Delta_i - \Lambda_i) - \eta_i = 0.$$ These can be written as $$-A_i^T\nu + \lambda_i\nabla\varphi_i(R_i+\gamma_i\Delta_i - \Lambda_i) \ge 0, \qquad A_i^T\nu - \gamma_i\lambda_i\nabla\varphi_i(R_i+\gamma_i\Delta_i - \Lambda_i) \ge 0,$$ which simplifies to .

# Example code

``` PythonPlus
import numpy as np
import cxvpy as cp

# We assume the following is already defined:
# - `A`, list of matrices A_i
# - `reserves`, list of vectors R_i
# - `fees`, list of fees for each CFMM i
# - `num_tokens`, list of n_i
# - `t`, amount of asset 1 provided to the network

current_assets = np.array([t, 0, 0])

# Build variables
deltas = [cp.Variable(n_i, nonneg=True) for n_i in num_tokens]
lambdas = [cp.Variable(n_i, nonneg=True) for n_i in num_tokens]
psi = cp.sum(A_i @ (L - D) for A_i, D, L in zip(A, deltas, lambdas))

# Objective is to trade t of asset 1 for a maximum amount of asset 3
obj = cp.Maximize(psi[2])

# Reserves after trade
new_reserves = [R + gamma_i*D - L \
    for R, gamma_i, D, L in zip(reserves, fees, deltas, lambdas)]

# Trading function constraints
cons = [
    # Balancer pool with weights 3, 2, 1
    cp.geo_mean(new_reserves[0], p=np.array([3, 2, 1])) \
        >= cp.geo_mean(reserves[0], p=np.array([3, 2, 1])),

    # Uniswap v2 pools
    cp.geo_mean(new_reserves[1]) >= cp.geo_mean(reserves[1]),
    cp.geo_mean(new_reserves[2]) >= cp.geo_mean(reserves[2]),
    cp.geo_mean(new_reserves[3]) >= cp.geo_mean(reserves[3]),

    # Constant sum pool
    cp.sum(new_reserves[4]) >= cp.sum(reserves[4]),
    new_reserves[4] >= 0,

    # Allow all assets at hand to be traded
    psi + current_assets >= 0
]

# Set up and solve problem
prob = cp.Problem(obj, cons)
prob.solve()

print(f"amount of asset 3 received: {psi[2].value}")
```


abstract: Artificial autonomous agents and robots interacting in complex environments are required to continually acquire and fine-tune knowledge over sustained periods of time. The ability to learn from continuous streams of information is referred to as lifelong learning and represents a long-standing challenge for neural network models due to catastrophic forgetting in which novel sensory experience interferes with existing representations and leads to abrupt decreases in the performance on previously acquired knowledge. Computational models of lifelong learning typically alleviate catastrophic forgetting in experimental scenarios with given datasets of static images and limited complexity, thereby differing significantly from the conditions artificial agents are exposed to. In more natural settings, sequential information may become progressively available over time and access to previous experience may be restricted. Therefore, specialized neural network mechanisms are required that adapt to novel sequential experience while preventing disruptive interference with existing representations. In this paper, we propose a dual-memory self-organizing architecture for lifelong learning scenarios. The architecture comprises two growing recurrent networks with the complementary tasks of learning object instances (episodic memory) and categories (semantic memory). Both growing networks can expand in response to novel sensory experience: the episodic memory learns fine-grained spatiotemporal representations of object instances in an unsupervised fashion while the semantic memory uses task-relevant signals to regulate structural plasticity levels and develop more compact representations from episodic experience. For the consolidation of knowledge in the absence of external sensory input, the episodic memory periodically replays trajectories of neural reactivations. We evaluate the proposed model on the CORe50 benchmark dataset for continuous object recognition, showing that we significantly outperform current methods of lifelong learning in three different incremental learning scenarios.
author: German I. Parisi$^1$, Jun Tani$^2$, Cornelius Weber$^1$, Stefan Wermter$^1$
$^1$`<!-- -->`{=html}Knowledge Technology, Department of Informatics, Universität Hamburg, Germany
$^2$`<!-- -->`{=html}Cognitive Neurorobotics Research Unit, Okinawa Institute of Science and Technology, Japan
title: Lifelong Learning of Spatiotemporal Representations with Dual-Memory Recurrent Self-Organization

# Introduction

Artificial autonomous agents and robots interacting in dynamic environments are required to continually acquire and fine-tune their knowledge over time (Thrun and Mitchell, 1995; Parisi et al., 2018a). The ability to progressively learn over a sustained time span by accommodating novel knowledge while retaining previously learned experiences is referred to as continual or lifelong learning. In contrast to state-of-the-art deep learning models that typically rely on the full training set being available at once (see LeCun et al., 2015 for a review), lifelong learning systems must account for situations in which the training data become incrementally available over time. Effective models of lifelong learning are crucial in real-world conditions where an autonomous agent cannot be provided with all the necessary prior knowledge to interact with the environment and the direct access to previous experience is restricted (Thrun and Mitchell, 1995). Importantly, there may be no distinction between training and test phases, which requires the system to concurrently learn and timely trigger behavioral responses (Cangelosi and Schlesinger, 2015; Tani, 2016). Lifelong machine learning represents a long-standing challenge due to catastrophic forgetting or interference, i.e., training a model with a new task leads to an abrupt decrease in the performance on previously learned tasks (McCloskey and Cohen, 1989). To overcome catastrophic forgetting, computational models must adapt their existing representations on the basis of novel sensory experience while preventing disruptive interference with previously learned representations. The extent to which a system must be flexible for learning novel knowledge and stable for preventing the disruption of consolidated knowledge is known as the stability-plasticity dilemma, which has been extensively studied for both computational and biological systems (e.g., Grossberg, 1980, 2007; Mermillod et al., 2013; Ditzler et al., 2015). Neurophysiological evidence suggests distributed mechanisms of structural plasticity that promote lifelong memory formation, consolidation, and retrieval in multiple brain areas (Power and Schlaggar, 2016; Zenke et al., 2017a). Such mechanisms support the development of the human cognitive system on the basis of sensorimotor experiences over sustained time spans (Lewkowicz, 2014). Crucially, the brain must constantly perform two complementary tasks: (i) recollecting separate episodic events (specifics), and (ii) learning the statistical structure from the episodic events (generalities). The complementary learning systems (CLS) theory (McClelland et al., 1995; Kumaran et al., 2016) holds that these two interdependent operations are mediated by the interplay of the mammalian hippocampus and neocortex, providing the means for episodic memory (specific experience) and semantic memory (general structured knowledge). Accordingly, the hippocampal system exhibits quick learning of sparse representations from episodic experience which will, in turn, be transferred and integrated into the neocortical system characterized by a slower learning rate with more compact representations of statistical regularities.

Re-training a (deep) neural architecture from scratch in response to novel sensory input can require extensive computational effort. Furthermore, storing all the previously encountered data in lifelong learning scenarios has the general drawback of large memory requirements. Instead, Robins (1995) proposed pseudo-rehearsal (or intrinsic replay) in which previous memories are revisited without the need of explicitly storing data samples. Pseudo-samples are drawn from a probabilistic or generative model and replayed to the system for memory consolidation. From a biological perspective, the direct access to past experiences is limited or restricted. Therefore, the replay of hippocampal representations in the absence of external sensory input plays a crucial role in memory encoding (Carr et al., 2011; Kumaran et al., 2016). Memory replay is argued to occur through the reactivation of neural patterns during both sleep and awake states (e.g., free recall; Gelbard-Sagiv et al., 2008). Hippocampal replay provides the means for the gradual integration of knowledge into neocortical structures through the reactivation of recently acquired knowledge interleaved with the exposure to ongoing episodic experience (McClelland et al., 1995). Consequently, the periodic replay of previously encountered samples can alleviate catastrophic forgetting during incremental learning tasks, especially when the number of training samples for the different classes is unbalanced or when a sample is encountered only once (Robins, 1995).

A number of computational approaches have drawn inspiration from the learning principles observed in biological systems. Machine learning models addressing lifelong learning can be divided into approaches that regulate intrinsic levels of plasticity to protect consolidated knowledge, that dynamically allocate neural resources in response to novel experience, or that use complementary dual-memory systems with memory replay (see section 2). However, most of these methods are designed to address supervised learning on image datasets of very limited complexity such as MNIST (LeCun et al., 1998) and CIFAR-10 (Krizhevsky, 2009) while not scaling up to incremental learning tasks with larger-scale datasets of natural images and videos (Kemker et al., 2018; Parisi et al., 2018a). Crucially, such models do not take into account the temporal structure of the input which plays an important role in more realistic learning conditions, e.g., an autonomous agent learning from the interaction with the environment. Therefore, in contrast to approaches in which static images are learned and recognized in isolation, we focus on lifelong learning tasks where sequential data with meaningful temporal relations become progressively available over time.

In this paper, we propose a growing dual-memory (GDM) architecture for the lifelong learning of spatiotemporal representations from videos, performing continuous object recognition at an instance level (episodic knowledge) and at a category level (semantic knowledge). The architecture comprises two recurrent self-organizing memories that dynamically adapt the number of neurons and synapses: the episodic memory learns representations of sensory experience in an unsupervised fashion through input-driven plasticity, whereas the semantic memory develops more compact representations of statistical regularities embedded in episodic experience. For this purpose, the semantic memory receives neural activation trajectories from the episodic memory and uses task-relevant signals (annotated labels) to modulate levels of neurogenesis and neural update. Internally generated neural activity patterns in the episodic memory are periodically replayed to both memories in the absence of sensory input, thereby mitigating catastrophic forgetting during incremental learning. We conduct a series of experiments with the recently published Continuous Object Recognition (CORe50) benchmark dataset (Lomonaco and Maltoni, 2017). The dataset comprises 50 objects within 10 categories with image sequences captured under different conditions and containing multiple views of the same objects (indoors and outdoors, varying background, object pose, and degree of occlusion). We show that our model scales up to learning novel object instances and categories and that it outperforms current lifelong learning approaches in three different incremental learning scenarios.

# Related Work

The CLS theory (McClelland et al., 1995) provides the basis for computational frameworks that aim to generalize across experiences while retaining specific memories in a lifelong fashion. Early computational attempts include French (1997) who developed a dual-memory framework using pseudorehearsal (Robins, 1995) to transfer memories, i.e., the training samples are not explicitly kept in memory but drawn from a probabilistic model. However, there is no empirical evidence showing that this or similar contemporaneous approaches (see O'Reilly and Norman, 2002 for a review) scale up to large-scale image and video benchmark datasets. More recently, Gepperth and Karaoguz (2015) proposed two approaches for incremental learning using a modified self-organizing map (SOM) and a SOM extended with a short-term memory (STM). We refer to these two approaches as GeppNet and GeppNet+STM, respectively. In GeppNet, task-relevant feedback from a regression layer is used to select whether learning in the self-organizing hidden layer takes place. In GeppNet+STM, the STM is used to store novel knowledge which is occasionally played back to the GeppNet layer during sleep phases interleaved with training phases. This latter approach yields better performance and faster convergence in incremental learning tasks with the MNIST dataset. However, the STM has a limited capacity, thus learning new knowledge can overwrite old knowledge. In both cases, the learning process is divided into the initialization and the actual incremental learning phase. Furthermore, GeppNet and GeppNet+STM require storing the entire training dataset during training. Kemker and Kanan (2018) proposed the FearNet model for incremental class learning inspired by studies of memory recall and consolidation in the mammalian brain during fear conditioning (Kitamura et al., 2017). FearNet uses a hippocampal network capable of immediately recalling new examples, a PFC network for long-term memories, and a third neural network inspired by the basolateral amygdala for determining whether the system should use the PFC or hippocampal network for a particular example. FearNet consolidates information from its hippocampal network to its PFC network during sleep phases. Kamra et al. (2018) presented a similar dual-memory framework for lifelong learning that uses a variational autoencoder as a generative model for pseudo-rehearsal. Their framework generates a short-term memory module for each new task. However, prior to consolidation, predictions are made using an oracle, i.e., they know which module contains the associated memory.

Different methods have been proposed that are based on regularization techniques to impose constraints on the update of the neural weights. This is inspired by neuroscience findings suggesting that consolidated knowledge can be protected from interference via changing levels of synaptic plasticity (Benna and Fusi, 2016) and is typically modeled in terms of adding regularization terms that penalize changes in the mapping function of a neural network. For instance, Li and Hoiem (2016) proposed a convolutional neural network (CNN) architecture in which the network that predicts the previously learned tasks is enforced to be similar to the network that also predicts the current task by using knowledge distillation, i.e., the transferring of knowledge from a large, highly regularized model to a smaller model. This approach, known as learning without forgetting (LwF), has the drawbacks of highly depending on the relevance of the tasks and that the training time for one task linearly increases with the number of old tasks. Kirkpatrick et al. (2017) proposed elastic weight consolidation (EWC) which adds a penalty term to the loss function and constrains the weight parameters that are relevant to retain previously learned tasks. However, this approach requires a diagonal weighting over the parameters of the learned tasks which is proportional to the diagonal of the Fisher information metric, with synaptic importance being computed offline and limiting its computational application to low-dimensional output spaces. Zenke et al. (2017b) proposed to alleviate catastrophic forgetting by allowing individual synapses to estimate their importance for solving a learned task. Similar to Kirkpatrick et al. (2017), this approach penalizes changes to the most relevant synapses so that new tasks can be learned with minimal interference. In this case, the synaptic importance is computed in an online fashion over the learning trajectory in the parameter space.

In general, regularization approaches comprise additional loss terms for protecting consolidated knowledge which, with a limited amount of neural resources, leads to a trade-off on the performance of old and novel tasks. Other approaches expand the neural architecture to accommodate novel knowledge. Rusu et al. (2016) proposed to block any changes to the network trained on previous knowledge and expand the architecture by allocating novel sub-networks with a fixed capacity to be trained with the new information. This prevents catastrophic forgetting but leads the complexity of the architecture to grow with the number of learned tasks. Draelos et al. (2017) trained an autoencoder incrementally using the reconstruction error to show whether the older digits were retained. Their model added new neural units to the autoencoder to facilitate the addition of new MNIST digits. Rebuffi et al. (2017) proposed the iCaRL approach which stores example data points that are used along with new data to dynamically adapt the weights of a feature extractor. By combining new and old data, they prevent catastrophic forgetting but at the expense of a higher memory footprint.

The approaches described above are designed for the classification of static images, often exposing the learning algorithm to training samples in a random order. Conversely, in more natural settings, we make use of the spatiotemporal structure of the input. In previous research (Parisi et al., 2017), we showed that the lifelong learning of action sequences can be achieved in terms of prediction-driven neural dynamics with internal representations emerging in a hierarchy of recurrent self-organizing networks. The networks can dynamically allocate neural resources and update connectivity patterns according to competitive Hebbian learning by computing the input based on its similarity with existing knowledge and minimizing interference by creating new neurons whenever they are required. This approach has shown competitive results with batch learning methods on action benchmark datasets. However, the neural growth and update are driven by the minimization of the bottomup reconstruction error and, thus, without taking into account top-down, task-relevant signals that can regulate the plasticitystability balance. Furthermore, the model cannot learn in the absence of external sensory input, which leads to a non-negligible degree of disruptive interference during incremental learning tasks.

# Proposed Method

The proposed architecture with growing dual-memory learning (GDM) comprises a deep convolutional feature extractor and two hierarchically arranged recurrent self-organizing networks (Figure 1). Both recurrent networks are extended versions of the Gamma-GWR model (Parisi et al., 2017) that dynamically create new neurons and connections in response to novel sequential input. The growing episodic memory (G-EM) learns from sensory experience in an unsupervised fashion, i.e., levels of structural plasticity are regulated by the ability of the network to predict the spatiotemporal patterns given as input. Instead, the growing semantic memory (G-SM) receives neural activation trajectories from G-EM and uses task-relevant signals (input annotations) to modulate levels of neurogenesis and neural update, thereby developing more compact representations of statistical regularities embedded in episodic experience. Therefore, G-EM and G-SM mitigate catastrophic forgetting through self-organizing learning dynamics with structural plasticity, increasing information storage capacity in response to novel input.

The architecture classifies image sequences at an instance level (episodic experience) and a category level (semantic knowledge). Thus, each input sample carries two labels which are used for the classification task at the different levels of the network hierarchy. For the consolidation of knowledge over time in the absence of sensory input, internally generated neural activity patterns in G-EM are periodically replayed to both memories, thereby mitigating catastrophic forgetting during incremental learning tasks. For this purpose, G-EM is equipped with synapses that learn statistically significant neural activity in the temporal domain. As a result, sequence-selective neural activation trajectories can be generated and replayed after each learning episode without explicitly storing sequential input.

## Gamma-GWR

The Gamma-GWR model (Parisi et al., 2017) is a recurrent extension of the Grow-When-Required (GWR) self-organizing network (Marsland et al., 2002) that embeds a Gamma memory (Principe et al., 1994) for representing short-term temporal relations. The Gamma-GWR can dynamically grow or shrink in response to the sensory input distribution. New neurons will be created to better represent the input and connections (synapses) between neurons will develop according to competitive Hebbian learning, i.e. neurons that activate simultaneously will be connected to each other. The Gamma-GWR learns the spatiotemporal structure of the input through the integration of temporal context into the computation of the self-organizing network dynamics.

The network is composed of a dynamic set of neurons, $A$, with each neuron consisting of a weight vector $\textbf{w}_j$ and a number $K$ of context descriptors $\textbf{c}_{j,k}$ ($\textbf{w}_j,\textbf{c}_{j,k}\in\mathbb{R}^n$). Given the input $\textbf{x}(t)\in\mathbb{R}^n$, the index of the best-matching unit (BMU), $b$, is computed as: $$\label{eq:GetB}
b = \arg\min_{j\in A}(d_j),$$ $$\label{eq:BMU}
d_j = \alpha_0 \Vert \textbf{x}(t) - \textbf{w}_j  \Vert^2 + \sum_{k=1}^{K}\ \alpha_k \Vert \textbf{C}_k(t)-\textbf{c}_{j,k}\Vert^2,$$ $$\label{eq:MergeStep}
\textbf{C}_{k}(t) = \beta \cdot \textbf{w}_b^{t-1}+(1-\beta) \cdot \textbf{c}_{b,k-1}^{t-1},$$ where $\Vert \cdot \Vert^2$ denotes the Euclidean distance, $\alpha_i$ and $\beta$ are constant factors that regulate the influence of the temporal context, $\textbf{w}_b^{t-1}$ is the weight vector of the BMU at $t-1$, and $\textbf{C}_{k}\in\mathbb{R}^n$ is the global context of the network with $\textbf{C}_{k}(t_0)=0$.

The activity of the network, $a(t)$, is defined in relation to the distance between the input and its BMU (Equation ) as follows: $$\label{eq:Activity}
a(t)=\exp(-d_b),$$ thus yielding the highest activation value of $1$ when the network can perfectly predict the input sequence (i.e. $d_b=0$). Furthermore, each neuron is equipped with a habituation counter $h_j \in [0,1]$ expressing how frequently it has fired based on a simplified model of how the efficacy of a habituating synapse reduces over time (Stanley, 1976). Newly created neurons start with $h_j=1$. Then, the habituation counter of the BMU, $b$, and its neighboring neurons, $n$, iteratively decrease towards 0. The habituation rule (Marsland et al., 2002) for a neuron $i$ is given by: $$\label{eq:FiringCounter}
\Delta h_i=\tau_i \cdot \kappa \cdot (1-h_i)-\tau_i,$$ with $i\in\{b,n\}$ and where $\tau_i$ and $\kappa$ are constants that control the monotonically decreasing behavior. Typically, $h_b$ is decreased faster than $h_n$ with $\tau_b>\tau_n$.

The network is initialized with two neurons and, at each learning iteration, a new neuron is created whenever the activity of the network, $a(t)$, in response to the input $\textbf{x}(t)$ is smaller than a given insertion threshold $a_T$. Furthermore, $h_b$ must be smaller than a habituation threshold $h_T$ in order for the insertion condition to hold, thereby fostering the training of existing neurons before new ones are added. The new neuron is created halfway between the BMU and the input. The training of the neurons is carried out by adapting the BMU $b$ and the neurons $n$ to which the $b$ is connected: $$\label{eq:UpdateRateW}
\Delta \textbf{w}_i = \epsilon_i \cdot h_i \cdot (\textbf{x}(t) - \textbf{w}_i),$$ $$\label{eq:UpdateRateC}
\Delta \textbf{c}_{i, k} = \epsilon_i \cdot h_i \cdot (\textbf{C}_k(t) - \textbf{c}_{i, k}),$$ with $i\in\{b,n\}$ and where $\epsilon_i$ is a constant learning rate ($\epsilon_n<\epsilon_b$). Furthermore, the habituation counters of the BMU and the neighboring neurons are updated according to Equation . Connections between neurons are updated on the basis of neural co-activation, i.e. when two neurons fire together (BMU and second-BMU), a connection between them is created if it does not yet exist. Each connection has an age that increases at each learning iteration. The age of the connection between the BMU and the second-BMU is reset to $0$, whereas the other ages are increased by a value of $1$. The connections with an age greater than a given threshold can be removed, and neurons without connections can be deleted.

For the purpose of classification, an associative matrix $H(j,l)$ stores the frequency-based distribution of sample labels during the learning phase so that each neuron $j$ stores the number of times that an input with label $l$ had $j$ as its BMU. Thus, the predicted label $\xi_j$ for a neuron $j$ can be computed as: $$\label{eq:WinnerLabel}
\xi_j = \arg\max_{l \in L} H(j,l),$$ where $l$ is an arbitrary label. Therefore, the unsupervised Gamma-GWR can be used for classification without requiring the number of label classes to be predefined.

## Episodic Memory

The learning process of growing episodic memory G-EM is unsupervised, thereby creating new neurons or updating existing ones to minimize the discrepancy between the sequential input and its neural representation. In this way, episodic memories can be acquired and fine-tuned iteratively through sensory experience. This is functionally consistent with hippocampal representations, e.g. in the dentate gyrus, which are responsible for pattern separation through the orthogonalization of incoming inputs supporting the auto-associative storage and retrieval of item-specific information from individual episodes (Yassa and Stark, 2011; Neuneubel and Knierim, 2014).

Given an input image frame, the extracted image feature vector (see section 4.1) is given as input to G-EM which recursively integrates the temporal context into the self-organizing neural dynamics. The spatial resolution of G-EM neurons can be tuned through the insertion threshold, $a_T$, with a greater $a_T$ leading to more fine-grained representations since new neurons will be created whenever $a(t)<a_T$ (see Equation ). The temporal depth is set by the number of context descriptors, $K$, with a greater $K$ yielding neurons that activate for larger temporal windows (longer sequences), whereas the temporal resolution is set by the hyperparameters $\alpha$ and $\beta$ (see Equation  and ).

To enable memory replay in the absence of external sensory input, we extend the Gamma-GWR model by implementing temporal connections that learn trajectories of neural activity in the temporal domain. Such temporal connections are sequence-selective synaptic links which are incremented between two consecutively activated neurons (Parisi et al., 2016). Sequence selectivity driven by asymmetric connections has been argued to be a feature of the cortex (Mineiro and Zipser, 1998), where an active neuron pre-activates neurons encoding future patterns. Formally, when the two neurons $i$ and $j$ are consecutively activated at time $t-1$ and $t$ respectively, their temporal synaptic link $P_{(i,j)}$ is increased by $\Delta P_{(i,j)} = 1$. For each neuron $i\in A$, we can retrieve the next neuron $v$ of a prototype trajectory by selecting $$\label{eq:Next}
v=\arg\max_{j \in A \setminus {i}} P_{(i,j)}.$$ Recursively generated neural activation trajectories can be used for memory replay (see section 3.4). During the learning phase, G-EM neurons will store instance-level label classes $\xi^I$ for the classification of the input (see Equation ). Furthermore, since trajectories of G-EM neurons are replayed to G-SM in the absence of sensory input, G-EM neurons will also store labels at a category label $l^C$. Therefore, the associative matrix for each neuron $j$ is of the form $H(j,l^I,l^C)$.

## Semantic Memory

The growing semantic memory G-SM combines bottom-up drive from neural activity in G-EM and top-down signals (i.e. category-level labels from the input) to regulate structural plasticity levels. More specifically, the mechanisms of neurogenesis and neural weight update are regulated by the ability of G-SM to correctly classify its input. Therefore, while G-EM iteratively minimizes the discrepancy between the input sequences and their internal representations, G-SM will create new neurons only if the correct label of a training sample cannot be predicted by its BMU in G-SM. This is implemented as an additional constraint in the condition for neurogenesis so that new neurons are not created unless the predicted label of the BMU (Equation ) does not match the input label.

G-SM receives as input activated neural weights from G-EM, i.e. the weight vector of a BMU in G-EM, $\textbf{w}_b^{\text{EM}}$, for a given input frame. As an additional mechanism to prevent novel sensory experience from interfering with consolidated representations, G-SM neurons are updated (Equation  and ) only if the predicted label for the BMU in G-SM matches in the input label, i.e. if the BMU codes for the same object category as the input. In this way, the representations of an object category cannot be updated in the direction of the input belonging to a different category, which would cause disruptive interference.

As a result of hierarchical processing, G-SM neurons code for information acquired over larger temporal windows than neurons in G-EM. That is, one G-SM will fire for a number $K^{\text{SM}}+1$ of neurons fired in G-EM (where $K^{\text{SM}}$ is the temporal depth of G-SM neurons). Since G-EM neurons will fire for a number $K^{\text{EM}}+1$ of input frames, G-SM neurons will code for a total of $K^{\text{SM}}+K^{\text{EM}}+1$ input frames. This is consistent with established models of memory consolidation where neocortical representations code for information acquired over more extended time periods than the hippocampus (e.g., Kumaran and McClelland, 2012; Kumaran et al., 2016), thereby yielding a higher degree of temporal slowness

Temporal slowness results from the statistical learning of spatiotemporal regularities, with neurons coding for prototype sequences of sensory experience. By using category-level signals to regulate neural growth and update, G-SM will develop more compact representations from episodic experience with neurons activating in correspondence of semantically-related input, e.g., the same neuron may activate for different instances of the same category and, because of the processing of temporal context, the same object seen from different angles. However, specialized mechanisms of slow feature analysis can be implemented that would yield invariance to complex input transformations such as view invariance (e.g., Berkes and Wiskott, 2005; Einhäuser et al., 2005). View invariance of objects is a prominent property of higher-level visual areas of the mammalian brain, with neurons coding for abstract representations of familiar objects rather than for individual views and visual features (Booth and Rolls, 1998; Karimi-Rouzbahani et al., 2017). Neurophysiological studies evidence that distributed representations in high-level visual regions of the neocortex (semantic) are less sparse than those of the hippocampus (episodic) and where related categories are represented by overlapping neural codes (Clarke and Tyler, 2014; Yamins et al., 2018).

## Memory Replay

Hippocampal replay provides the means for the gradual integration of knowledge into neocortical structures and is thought to occur through the reactivation of recently acquired knowledge interleaved with the exposure to ongoing experiences (McClelland et al., 1995). Although the periodic replay of previous data samples can alleviate catastrophic forgetting, storing all previously encountered data samples has the general drawback of large memory requirements and large retraining computational times.

In pseudo-rehearsal (or intrinsic replay), memories are drawn from a probabilistic or generative model and replayed to the system for memory consolidation (Robins, 1995). In our case, however, we cannot simply draw or generate isolated and randomly selected pseudo-samples from a given distribution since we must account for preserving the temporal structure of the input. Therefore, we generate pseudo-patterns in terms of temporally-ordered trajectories of neural activity. For this purpose, we propose to use the asymmetric temporal links of G-EM (section 3.2) to recursively reactivate sequence-selective neural activity trajectories (RNATs) embedded in the network. RNATs can be computed for each neuron in G-EM for a given temporal window and replayed to G-EM and G-SM after each learning episode triggered by external input stimulation.

For each neuron $j$ in G-EM, we generate a RNAT, $S_j$, of length $\lambda=K^{\text{EM}}+K^{\text{SM}}+1$ as follows: $$\label{eq:RNATs}
S_j=\langle \textbf{w}^{\text{EM}}_{\text{s}(0)},\textbf{w}^{\text{EM}}_{\text{s}(1)},...,\textbf{w}^{\text{EM}}_{\text{s}(\lambda)} \rangle,$$ $$\label{eq:RNATs1}
\text{s}(i) = \arg\max_{n \in A \setminus {j}} P_{(n,\text{s}(i-1))}, i \in [1,\lambda],$$ where $P_{(i,j)}$ is the matrix of temporal synapses (as defined by Equation ) and $\text{s}(0)=j$. The class labels of the pseudo-patters in $S_j$ can be retrieved according to Equation .

The set of generated RNATs from all G-EM neurons is replayed to G-EM and G-SM after each learning episode, i.e., a learning epoch over a batch of sensory observations. As a result of computing RNATs, sequence-selective prototype sequences can be generated and periodically replayed without the need of explicitly storing the temporal relations and labels of previously seen training samples. This is conceptually consistent with neurophysiological studies evidencing that hippocampal replay consists of the reactivation of previously stored patterns of neural activity occurring predominantly after an experience (Kudrimoti et al., 1999; Karlsson and Frank, 2009).

# Experimental Results

We perform a series of experiments evaluating the performance of the proposed GDM model in batch learning (section 4.2), incremental learning (section 4.3), and incremental learning with memory replay (section 4.4). We analyze and evaluate our model with the CORe50 dataset (Lomonaco and Maltoni, 2017; see section 4.1), a recently published benchmark for continuous object recognition from video sequences. We reproduce three experimental conditions defined by the CORe50 benchmark (section 4.5) showing that our model significantly outperforms state-of-the-art lifelong learning approaches. For the replication of these experiments, the source code of the GDM model is available as a repository.[^1]

## Feature Extraction

The CORe50 comprises 50 objects within 10 categories with image sequences captured under different conditions and multiple views of the same objects (varying background, object pose, and degree of occlusion; see Figure ). Each object comprises a video sequence of approximately 15 seconds where the object is shown to the vision sensor held by a human operator. The video sequences were collected in 11 sessions (8 indoors, 3 outdoors) with a Kinect 2.0 sensor delivering RGB ($1027 \times 575$) and depth images ($512 \times 242$) at 20 frames per second (fps) for a total of 164,866 frames. For our experiments, we used $128 \times 128$ RGB images provided by the dataset at a reduced frame rate of 5hz. The movements performed by the human operator with the objects (e.g. rotation) are quite smooth and reducing the number of frames per second has not shown significant loss of information.

For a more direct comparison with the baseline results provided by Lomonaco and Maltoni (2017) who adopted the VGG model (Simonyan and Zisserman, 2014) pre-trained on the ILSVRC-2012 dataset (Russakovsky et al., 2014), our feature extraction module consists of the same pre-trained VGG model to which we applied a convolutional operation with 256 1x1 kernels on the output of the fully-connected hidden layer to reduce its dimensionality from 2048 to 256. Therefore, G-EM receives a 256-dimensional feature vector per sequence frame. Such compression of the feature vectors is desirable since the Gamma-GWR uses the Euclidean distance as a metric to compute the BMUs, which becomes weakly discriminant when the data are very high-dimensional or sparse (Parisi et al., 2015). Furthermore, it is expected that different pre-trained models may exhibit a slightly better performance than VGG, e.g. ResNet-50 (He et al., 2016; see Lomonaco and Maltoni, 2018 for ResNet-50 performance on CORe50). However, here we focus on showing the contribution of context-aware growing networks rather than comparing deep feature extractors.

## Batch Learning

We trained the architecture on the entire training data at once and subsequently tested its classification performance at instance and category level. Following the same evaluation scheme described by Lomonaco and Maltoni (2017), we used the samples from sessions $\#3$, $\#7$, $\#10$ for testing and the samples from the remaining 8 sessions for training. We compare our results to the baseline provided by Lomonaco and Maltoni (2017) using fine-tuning on a pre-trained VGG network (VGG+FT). To better assess the contribution of temporal context (TC) for the task of continuous object recognition, we performed batch learning experiments with 3 different model configurations:

-   **GDM**: We trained the model using TC and tested it on novel sequences. For each input frame, an object instance and an object category are predicted.

-   **GDM (No TC)**: We trained and tested the model without TC by setting $K=0$, i.e. the computation of the BMU is reduced to $b = \arg\min_{j\in A}\Vert \textbf{x}(t) - \textbf{w}_j  \Vert^2$.

-   **GDM (No TC during test)**: We trained the model with TC but tested on single image frames by setting $K=0$ during the test phase.

The training hyperparameters are listed in Table . Except for the insertion thresholds $a^{\text{EM}}_T$ and $a^{\text{SM}}_T$, the remaining parameters were set similar to Parisi et al. (2017) for the incremental learning of sequences. Larger insertion thresholds will lead to a larger number of neurons. However, the best classification performance will not be necessarily obtained by the largest number of neurons. In G-EM, the neural representation should be characterized by a sufficiently high spatiotemporal resolution for discriminating between similar object instances and replaying episodic experience in the absence of sensory input. Conversely, regulated unsupervised learning in G-SM will lead to a more compact, overlapping neural representation with a smaller number of neurons while preserving the ability to correctly classify its input. The number of context descriptors ($K^{\text{EM}}$, $K^{\text{SM}}$) is set to $2$. This means that G-EM neurons will activate in correspondence of 3 image frames and G-SM neurons in correspondence of 3 G-EM neurons, i.e. a processing window of 5 frames (1s of video at 5fps). Additional experiments showed that increasing the number of context descriptors does not significantly improve the overall accuracy. This is because a small number of context descriptors will lead to learning short-term temporal relations which are useful for temporal slowness, i.e. neurons that activate for multiple similar views of the same object (where different views of the object are induced by object motion). Neurons with a higher temporal depth will learn longer-term temporal relations and, depending on the difference between the training and test set, training with longer sequences may result in the specialization of neurons to the sequences in the training set while failing to generalize. Therefore, convenient values for $K^{\text{EM}}$ and $K^{\text{SM}}$ can be selected according to different criteria and properties of the input, e.g. number of frames per second, smoothness of object motion, desired degree of neural specialization.

The classification performance for the 3 different configurations is summarized in Table , showing instance-level and category-level accuracy after 35 training epochs averaged across 5 learning trials in which we randomly shuffled the batches from different sessions. The best results were obtained by GDM using temporal context with an average accuracy of 79.43% (instance level) and 93.92% (category level), showing an improvement of 10.35% and 13.69% respectively with respect to the baseline results (Lomonaco and Maltoni, 2017). Without the use of temporal context, the accuracy is comparable to the baseline showing a marginal improvement of 1.34% (instance level) and 3.31% (category level). Our results demonstrate that learning the temporal relations of the input plays an important role for this dataset. Interestingly, dropping the temporal component during the test phase, i.e. using single image frames for testing on context-aware networks, shows a slightly better performance (2.14% and 3.78% respectively) than training without temporal context. This is because trained neural weights embed some temporal structure of the training sequences and, consequently, the context-free computation of a BMU from a single input frame will still be matched to context-aware neurons.

| **Hyperparameters** | **Value** |  |  |  |  |
|:---|:---|:--:|:--:|:--:|:--:|
| Insertion thresholds | $a^{\text{EM}}_T=0.3$, $a^{\text{SM}}_T=0.001$ |  |  |  |  |
| Habituation counters | $h_T=0.1$, $\tau_{b}=0.3$, $\tau_{n}=0.1$, $\kappa=1.05$ |  |  |  |  |
| Temporal depth | $K^{\text{EM}}= 2$, $K^{\text{SM}}= 2$ |  |  |  |  |
| Temporal context | $\alpha=[0.67,0.24,0.09]$, $\beta=0.7$ |  |  |  |  |
| Learning rates | $\epsilon_b=0.5$, $\epsilon_n=0.005$ |  |  |  |  |

Training hyperparameters for the G-EM and G-SM networks (batch and incremental learning).

|  |  |  |  |  |  |
|:---|:--:|:--:|:--:|:--:|:--:|
|  | **Accuracy (%)** | **Accuracy (%)** |  |  |  |
| **Approach** | (Instances) | (Categories) |  |  |  |
| VGG + FT (Lomonaco and Maltoni, 2017) | 69.08 | 80.23 |  |  |  |
| Proposed GDM (No TC) | 70.42 | 83.54 |  |  |  |
| Proposed GDM (No TC during test) | 72.56 | 87.32 |  |  |  |
| Proposed GDM | **79.43** | **93.92** |  |  |  |

Comparison of batch learning performance for instance-level and category-level classification. We show the accuracy for the pre-trained VGG with fine-tuning (VGG+FT) and the proposed GDM for three different configurations: (i) growing networks with temporal context (TC), (ii) without TC, and (iii) without TC during test. Best results in bold.

Figure  shows the number of neurons, update rate, and classification accuracy for G-EM and G-SM (with temporal context) through 35 training epochs averaged across 5 learning trials. It can be seen that the average number of neurons created in G-EM is significantly higher than in G-SM (Figure .A). This is expected since G-EM will grow to minimize the discrepancy between the input and its internal representation, whereas neurogenesis and neural update rate in G-SM are regulated by the ability of the network to predict the correct class labels of the input. The update rate (Figure .B) is given by multiplying the fixed learning rate by the habituation counter of the neurons ($\epsilon_i \cdot h_i$), which shows a monotonically decreasing behavior. This indicates that, after a number of epochs, the created neurons become habituated to the input. Such a habituation mechanism has the advantage of protecting consolidated knowledge from being disrupted or overwritten by the learning of novel sensory experience, i.e. well-trained neurons will respond slower to changes in the distribution and the network will create new neurons to compensate for the discrepancy between the input and its representation.

## Incremental Learning

In the incremental learning strategy, the training samples of different object categories become progressively available over time, i.e., each mini-batch contains all the instances of an object category from all the 8 training sessions. Each category batch is shown once to the model and samples from that category are not shown again during the learning of new categories. Therefore, the model must incrementally learn new object instances and categories without forgetting previously learned ones. For a direct comparison with our previous experiment, the hyperparameters for the incremental learning experiment are the same as for the batch learning strategy (Table ).

Figure  shows the number of neurons, update rate, and accuracy over 10 epochs (i.e. the 10 object categories) averaged across 5 runs of randomly shuffled object categories. The variance from the mean values (shaded areas in Figure ) shows that the order of exposure to object categories can affect the final result. In general, the number of neurons increases over time (Figure .A) and, in contrast to the batch learning strategy where neurogenesis is particularly strong during the initial training epochs (Figure .A), in this case new neurons are progressively created in response to the exposure of the model to novel object classes. Similarly, the update rate for both networks (Figure .B) does not monotonically decrease over time but rather stays quite stable in correspondence to novel sensory experience. Since newly created neurons are not well trained, the update rate will be higher at the moment of neural insertion and progressively decrease as the newly created neurons become habituated. The overall accuracy decreases with the number of object categories encountered, showing a higher sensitivity of the model with respect to the order in which the object categories are presented (Figure .C). The average classification accuracy for the incremental learning strategy is $75.93\% \pm 2.23$ (instance level) and $85.53\% \pm 1.35$ (category level), showing a decrease of $3.5\%$ and $8.39\%$ respectively compared to the batch learning performance. This suggests that an additional mechanism such as memory replay is required to prevent the disruptive interference of existing representations.

Figure  shows a comparison of the effects of forgetting during the incremental learning strategy in terms of the overall accuracy on the categories encountered so far and the accuracy on the first encountered category as new categories are learned. For the object instances, we compare the overall accuracy (Figure .A) with the accuracy on the first 5 encountered instances (i.e. 1 category; see Figure .B), showing that for the latter the accuracy drops to $69.25\% \pm 4.31$ (compared to $75.93\% \pm 2.23$). For the object categories (Figure .C-D), the accuracy on the first encountered category drops to $79.53\% \pm 5.23$ (compared to $85.53\% \pm 1.35$). Overall, these results suggest that memory replay is an important feature for the reactivation of previously learned neural representations at the moment of learning from novel sensory experience with the goal to prevent that classes that have been encountered at early stages be forgotten over time.

## Incremental Learning with Memory Replay

In this learning strategy, we trained the model as described above with progressively available mini-batches containing 1 object category each. Here, however, after each learning episode (i.e. a training epoch over the mini-batch), the model generates a set of RNATs, $S_j$ (Equation  and ) from the G-EM neurons. Thus, the number of RNATs of length $\lambda=5$ is equal to the number of neurons created by G-EM. The set of RNATs is replayed to G-EM and G-SM in correspondence of novel sensory experience to reinforce previously encountered categories. Since the growing self-organizing networks store the global temporal context, $\textbf{C}_{k}(t)$, over the training iterations (Equation ) for learning the temporal structure of the input, each RNAT is fed into G-EM and G-SM as a single sample batch and the global temporal context is reset to zero after one epoch. It is expected that, by periodically replaying RNATs when new categories are encountered, knowledge representations will consolidate over time and, consequently, significantly alleviate catastrophic forgetting.

The benefit of using memory replay is shown in Figure  where we compare the overall accuracy on all the categories encountered so far to the accuracy on the first encountered category over the number of encountered categories. At an instance level (Figure .A-B), incremental learning with memory replay improves the overall accuracy to $82.14\% \pm 2.05$ (from $75.93\% \pm 2.23$) and accuracy on the first 5 instances to $80.41\% \pm 1.35$ (from $69.25\% \pm 2.01$). At a category level (Figure .C-D), the overall accuracy increases to $91.18\% \pm 0.25$ (from $85.53\% \pm 1.35$) and the accuracy on the first encountered category to $89.21\% \pm 3.37$ (from $79.53\% \pm 5.23$) Overall, our results support the hypothesis that replaying RNATs generated from G-EM mitigates the effects of catastrophic forgetting.

## Continuous Object Recognition

We evaluate our model with the 3 incremental learning scenarios proposed by the CORe50 benchmark for the task of continuous object recognition:

**New Instances (NI)**: New instances of the same class and from different acquisition sessions become progressively available and are shown once to the model. Therefore, all the classes to be learned are known. For all the classes, the model is trained with the instances of a first session and subsequently with the remaining 7 sessions. (Here, the term *classes* is used for object categories.)

**New Classes (NC)**: Training samples from novel different classes become available over time, thus the model must deal with the learning of new classes without forgetting previously learned ones. Each training batch contains all the sequences of a small group of classes and memory replay is possible across batches. The first batch includes 10 objects while the remaining 8 batches contain 5 objects each. The test set includes samples from all the classes and the model is required to classify samples that have not been seen yet (except for the last evaluation step).

| **Method** | **Avg. Acc. (%)** | **Std. Dev. (%)** |  |  |  |
|:---|:---|:--:|:--:|:--:|:--:|
| *New Instances (NI)* |  |  |  |  |  |
| Proposed GDM (with replay) | **87.94** | 1.72 |  |  |  |
| Proposed GDM | 74.87 | 2.54 |  |  |  |
| Cumulative (Lomonaco and Maltoni, 2017) | 65.15 | 0.66 |  |  |  |
| LwF (Zhizhong and Hoiem, 2016) | 59.42 \* | 2.71 |  |  |  |
| EWC (Kirkpatrick et al., 2017) | 57.40 \* | 3.80 |  |  |  |
| Naïve (Lomonaco and Maltoni, 2017) | 54.69 | 6.18 |  |  |  |
| *New Classes (NC)* |  |  |  |  |  |
| Proposed GDM (with replay) | **86.14** | 2.03 |  |  |  |
| Proposed GDM | 73.02 | 2.91 |  |  |  |
| Cumulative | 64.65 | 1.04 |  |  |  |
| iCaRL (Rebuffi et al., 2016) | 43.62 \* | 0.66 |  |  |  |
| CWR (Lomonaco and Maltoni, 2017) | 42.32 | 1.09 |  |  |  |
| LwF | 27.60 \* | 1.70 |  |  |  |
| EWC | 26.22 \* | 1.18 |  |  |  |
| Naïve | 10.75 | 0.84 |  |  |  |
| *New Instances and Classes (NIC)* |  |  |  |  |  |
| Proposed GDM (with replay) | **87.06** | 2.13 |  |  |  |
| Proposed GDM | 72.57 | 2.96 |  |  |  |
| Cumulative | 64.13 | 0.88 |  |  |  |
| CWR | 29.56 | 0 |  |  |  |
| LwF | 28.94 \* | 4.30 |  |  |  |
| EWC | 28.31 \* | 4.30 |  |  |  |
| Naïve | 19.39 | 2.90 |  |  |  |

Accuracy on the CORe50 incremental learning scenarios. Results denoted with \* indicate the re-implementation of the method by Lomonaco and Maltoni (2017). Best results in bold.

**New Instances and Classes (NIC)**: New instances and classes become available over time, requiring the model to consolidate knowledge about known classes and to learn new ones. The first batch includes 10 classes and the subsequent batches 5 classes each, with only one training sequence per class included in the batches. This scenario comprises 79 batches, maximizing the categorical representation in the first batch and randomly selecting the remaining 78 batches.

For each scenario, we compute the average accuracy over 10 configurations of randomly shuffled batches. The results for the NI, NC, and NIC scenarios compared to other approaches are listed in Table  . It can be seen that our proposed method with memory replay produces state-of-the-art results for this benchmark dataset, showing an average accuracy of 87,94%, 86.14%, and 87.06% for the NI, NC, and NIC scenarios, respectively. These results represent a large increase in accuracy over 20% for each scenario with respect to the previous best results, i.e., a cumulative approach reported by Lomonaco and Maltoni (2017). The authors reported results using 3 methods with pre-trained CNN models and $128\times128$ images: (i) a naïve approach which consists of continuous stochastic gradient descent training as new batches become available, (ii) a proposed *CopyWeights with Re-init* (CWR) method that skips layers *fc6* and *fc7* of the CNN (for details, see Lomonaco and Maltoni, 2017; page 7), and (iii) a cumulative approach where the learning is carried out by considering the current batch and all the previous ones.

Ours and previously reported experiments show that lifelong learning is a very challenging task and that the overall performance of some approaches can differ significantly according to the specific learning strategy. Furthermore, a more direct comparison of the model's behavior is hindered by the fact that the other methods do not comprise recurrent neural dynamics that account for learning the temporal structure of the input which, in this case, is a clear advantage (see Table ) since the temporal relations of the input can be exploited for more robust learning and prediction.

The experiments reported for all the 3 incremental learning scenarios were conducted with the test set containing samples from all the seen classes (except for the last evaluation step). Such an evaluation scheme was selected to keep the test set consistent across all the scenarios (Lomonaco and Maltoni, 2017). However, in a more realistic lifelong learning scenario, the model should be able to deal with unknown classes during sequence retrieval. In our case, the model will always predict an output label in correspondence to a retrieved sequence. Instead, it would be convenient to design a novelty detection mechanism for unseen classes so that the system will output a predicted label provided that the input sequence produces a sufficient level of neural activity (Parisi et al., 2015).

# Discussion

## Summary

We proposed a growing dual-memory architecture with self-organizing networks for the lifelong learning of spatiotemporal representations from image sequences. The GDM model can perform continuous object recognition at an instance level (episodic experience) and at a category level (semantic knowledge). We introduced the use of recurrent self-organizing networks, in particular of extended versions of the GammaGWR, to model the interplay of two complementary learning systems: an episodic memory, G-EM, with the task of learning fine-grained spatiotemporal representations from sensory experience and a semantic memory, G-SM, for learning more compact representations from episodic experience. With respect to previously proposed dual-memory learning systems, our contribution is threefold. First, in contrast to the predominant approach of processing static images independently, we implement recurrent self-organizing memories for learning the spatiotemporal structure of the input. Second, as a complementary mechanism to unsupervised growing networks, we use task-relevant signals to regulate structural plasticity levels in the semantic memory, leading to the development of more compact representations from episodic experience. Third, we model memory replay as the periodic reactivation of neural activity trajectories from temporal synaptic patterns embedded in an episodic memory. Our experiments show that the proposed GDM model significantly outperforms state-of-the-art lifelong learning methods in three different incremental learning tasks with the CORe50 benchmark dataset.

## Growing Recurrent Networks with Memory Replay

The use of growing networks leads to the dynamic allocation of additional neurons and connections in response to novel sensory experience. In particular, the Gamma-GWR (Parisi et al., 2017) provides the basic mechanism for growing self-organizing memories with temporal context for learning the spatiotemporal structure of the input in an unsupervised fashion. Different models of neural network self-organization have been proposed that resemble the dynamics of Hebbian learning and plasticity (Fritzke, 1995; Kohonen, 1995; Marsland et al., 2002), with neural map organization resulting from unsupervised statistical learning. For instance, in the traditional self-organizing feature map and its dynamic variant (e.g., Kohonen, 1995; Rougier and Boniface, 2011), the number of neurons is pre-defined.

Empirically selecting a convenient number of neurons can be tedious for networks with recurrent dynamics, especially when dealing with non-stationary input distributions (Strickert and Hammer, 2005). To alleviate this issue, growing self-organizing networks for temporal processing have been proposed, for instance the Gamma-GNG (Estévez and Vergara, 2012) that equips neurons with a temporal context. However, the GammaGNG grows at a constant, pre-defined interval and does not consider whether previously created neurons have been well trained before creating new ones. This will lead to scalability issues if the selected interval is too short or, conversely, to an insufficient number of neurons if the interval is too large. Therefore, we extended the Gamma-GWR which can quickly react to changes in the input distribution and can create new neurons whenever they are required.

From a biological perspective, there has been controversy over whether in human adults detectable amounts of new neurons can grow. Recent research has suggested that hippocampal neurogenesis drops sharply in children (Sorrells et al., 2018) and becomes undetectable in adulthood, whereas other studies suggest that hippocampal neurogenesis sustains human-specific cognitive function throughout life (Boldrini et al., 2018). Neurophysiological studies evidence that, in addition to neurogenesis, synaptic rewiring by structural plasticity has a significant contribution on memory formation in adults (Knoblauch et al., 2014; Knoblauch, 2017), with a major role of structural plasticity in increasing information storage efficiency in terms of space and energy demands. While the mechanisms for creating new neurons and connections in the Gamma-GWR do not resemble biologically plausible mechanisms of neurogenesis and synaptogenesis (e.g., Eriksson et al., 1998; Ming and Song, 2011; Knoblauch, 2017), the GWR learning algorithm represents an efficient computational model that incrementally adapts to non-stationary input. Crucially, the GWR model creates new neurons whenever they are required and only after the training of existing ones. The neural update rate decreases as the neurons become more habituated, which has the effect of preventing that noisy input interferes with consolidated neural representations. Alternative theories suggest that an additional function of hippocampal neurogenesis is the encoding of time for the formation of temporal associations in memory (Aimone et al., 2006, 2009), e.g., in terms of temporal clusters of long-term episodic memories. This represents an interesting research direction for the modeling of temporal associations in the Gamma-GWR.

For mitigating catastrophic forgetting during incremental learning tasks, the proposed model generates recurrent neural activity trajectories (RNATs; Equation 10) after each learning episode. The set of generated RNATs is periodically replayed to both networks in correspondence of novel sensory experience for the consolidation of knowledge over time. This is consistent with biological evidence suggesting that the reactivation of hippocampal representations and their frequent replay to the neocortex are crucial for memory consolidation and retrieval (see Carr et al., 2011 for a review). The process of replaying previously seen data without explicitly storing data samples is referred to as intrinsic replay (Robins, 1995) and has the advantage of fewer memory requirements with respect to explicitly storing training samples. In our approach, the episodic memory G-EM embeds the temporal structure of the input through the implementation of temporal synapses that are strengthened between consecutively activated neurons (Equation 9). Therefore, RNATs comprise prototype sequence snapshots that can be generated without the need of explicitly storing the training sequences. Our reported results show that the use of RNATs yields a significantly improved overall accuracy during incremental learning.

In this work, we have focused on regulating the mechanisms of neurogenesis and neural update, whereas we have not investigated the removal of old connections and isolated neurons. At each learning iteration of the Gamma-GWR, old connections exceeding a given age threshold and neurons without connections can be deleted. Removing a neuron from the network means that the knowledge coded by that unit is permanently forgotten. Therefore, a convenient maximum age of connections must be set to avoid catastrophic forgetting. In incremental learning scenarios, it is non-trivial to define a convenient age threshold for connections to be removed since data samples become available over time and neurons coding for consolidated knowledge might not fire for a large number of iterations. Mechanisms of intrinsic memory replay as modeled in this paper could be used to prevent the deletion of consolidated knowledge. For instance, the periodic replay of episodic representations would prevent the networks from deleting relevant knowledge also when external sensory input does not activate those representations for sustained periods of time.

Conceptual similarities can be found between our model and the adaptive resonance theory (ART) in which neurons are iteratively adapted to a non-stationary input distribution in an unsupervised fashion and new neurons can be added in correspondence of dissimilar input (see Grossberg, 2012 for a review). The primary intuition of the ART model is that learning occurs via the interaction of top-down and bottom-up processes, where top-down expectations act as memory templates or prototypes which are compared to bottom-up sensory observations. Similar to the GWR's activation threshold, the ART model uses a vigilance parameter to produce fine-grained or more general memories. Despite its inherent ability to mitigate catastrophic forgetting during incremental learning, it has been noted that the results of some variants of the ART model depend significantly upon the order in which the training data are processed. However, an extensive evaluation with recent lifelong learning benchmarks has not been reported. Therefore, ART-based models represent an additional complementary approach to growing self-organizing models.

## Conclusion

Lifelong learning represents a fundamental but challenging component of artificial systems and autonomous agents. Despite significant advances in this direction, current models of lifelong learning are far from providing the flexibility, robustness, and scalability exhibited by biological systems. In this paper, we contribute to extending dual-memory models for the processing of sequential input which represents more realistic experimental settings compared to learning from static image datasets. In the future, it would be interesting to extend this model to the multisensory domain, e.g., where neural representations can be continually learned from audio-visual streams (Parisi et al., 2016, 2018b). The proposed architecture can be considered a further step toward more flexible lifelong learning methods that can be deployed in embodied agents for incrementally acquiring and refining knowledge over sustained periods through the active interaction with the environment.

# Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

# Acknowledgments

This research was partially supported by the German Research Foundation (DFG) under project Transregio Crossmodal Learning (TRR 169). The authors would like to thank Pablo Barros and Vadym Gryshchuk for technical support.

[^1]: GDM model: https://github.com/giparisi/GDM