# Tensor-based approach for training flexible neural networks

Yassine Zniyed, Konstantin Usevich, Sebastian Miron, David Brie

► To cite this version:

Yassine Zniyed, Konstantin Usevich, Sebastian Miron, David Brie. Tensor-based approach for training flexible neural networks. 55th Asilomar Conference on Signals, Systems and Computers, Oct 2021, Pacific Grove, CA, United States. pp.1673-1677, 10.1109/IEEECONF53345.2021.9723324. hal-03518648

HAL Id: hal-03518648

https://hal.science/hal-03518648v1

Submitted on 27 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# A tensor-based approach for training flexible neural networks

Yassine Zniyed*, Konstantin Usevich†, Sebastian Miron†, and David Brie†

*Université de Toulon, Aix-Marseille Université, CNRS, LIS, Toulon, France

Email: yassine.zniyed@univ-tln.fr

†CRAN, Université de Lorraine, CNRS, Vandœuvre-lès-Nancy, France

Email: firstname.lastname@univ-lorraine.fr

***Abstract***—Despite excellent prediction performance, state-of-the-art neural network (NN) architectures are very large, up to several millions of weights. In particular, running them on systems with limited computational capacity (embedded systems) becomes a difficult task. In this work, we focus on the compression of NNs, considering a tensor-based approach. In particular, we are interested in the estimation of flexible activation functions using tensor-based solutions. The proposed method is based on a coupled matrix-tensor factorization (CMTF), fusing the zeroth and first order information of the NN. The proposed learning algorithm is based on a constrained alternating least squares (ALS) approach. The activation functions (AFs) are expressed as a weighted sum of pre-defined basis elements. Our method allows to strongly compress large NN layers, with a very slight degradation of the classification accuracy. This also provides insights into the functioning of NNs, which is a problem of utmost interest in NN domain.

## I. INTRODUCTION

Neural networks (NNs) are powerful tools, which have the ability to learn and model complex nonlinear functions, while showing good prediction performances. Before training a neural network, one has to decide (i) how many neurons and hidden layers to use? and (ii) which activation functions (AFs) to employ? These choices control how well the NN learns from the data. State-of-the-art modern neural network architectures are too large and have a lot of parameters. Running them in systems with limited computational capacity is a difficult task. For this reason, several works in the literature have been interested in the compression of these models.

Most popular tensor approaches [1] for compression mainly aim at compressing the layers of convolutional networks, which can be viewed as tensors. By a canonical polyadic decomposition (CPD) [2], [3], they replace multidimensional convolutions by one-dimensional ones. Another direction of research is focused on relating tensor decompositions to neural networks with product units (instead of summing units) [4]; this type of representations, however, is not so much used in practice. In this work, we consider an entirely different approach. While keeping the traditional neural network structure (linear weights + nonlinear activation functions), we aim at adding flexibility [5] to activation functions (AFs), as opposed to fixed AFs used conventionally. In particular, the activation functions are allowed to be different in different nodes. Such architecture is particularly interesting thanks to

identifiability (uniqueness) theory available in the polynomial case [6]. Identifiability properties may provide insight into the functioning of these NNs and help to enforce stability of the representation.

Unlike existing methods for flexible AFs (FAFs) that are using conventional training techniques [5], we employ an original framework developed in nonlinear system identification community. The work of [7] showed that an architecture with one hidden flexible layer can be identified as a CPD of a Jacobian tensor. However, it is not directly applicable in the learning setup; in particular, there is no simple way to estimate the activation functions. In this work we propose a new method for compression of pretrained neural networks based on Coupled Matrix-Tensor Factorization (CMTF). This technique fuses the first and zeroth order information of the NN, where the first-order information is contained in a Jacobian tensor, following a constrained CPD.

The new flexible NN model has different and flexible AFs in each neuron. The FAFs are expressed as weighted sum of predefined basis functions. The purpose of this model is to increase the expressiveness of the model to be able to learn functions with lower parameters compared to the classical models. This kind of result is likely to speed up the execution of neural networks and will allow their use on systems with low computing resources.

***Notations:*** The outer product is denoted by $\otimes$. Tensors are represented by bold calligraphic capital letters, e.g., $\mathcal{X}$. The norm of a tensor is denoted by $||\cdot||$. The operator diag(·) forms a diagonal matrix from its vector argument. unfold$_q$ $\mathcal{X}$ refers to the unfolding of tensor $\mathcal{X}$ over its q-th mode [8].

## II. FLEXIBLE ACTIVATION FUNCTION MODEL

### A. Flexible model

The model proposed in this work is a flexible NN model, where each neuron has a different and flexible AF. The AFs are expressed as a linear combination of predefined basis functions. A graphical illustration of a basic one-layer block of this model is given in Fig. 1.

The multivariate function that represents the input-output relationship is denoted by $f$. This function $f: \mathbb{R}^m \rightarrow \mathbb{R}^n$ is
<figure><img src="image_2.png" /><figcaption>Fig. 1. Graphical representation of the basic one-layer block of flexible NNs.</figcaption></figure>

vector valued, that is:

$$ \mathbf{f}(\mathbf{u}) = [f_1(\mathbf{u}) \cdots f_n(\mathbf{u})]^{\top}, \quad \text{with} \quad \mathbf{u} = [u_1 \cdots u_m]^{\top}. \quad (1) $$

We say that **f** admits the flexible NN model if it can be expressed as:

$$ \mathbf{f}(\mathbf{u}) = \mathbf{W}\mathbf{g}(\mathbf{V}^{\top}\mathbf{u}), \quad (2) $$

where $\mathbf{V} \in \mathbb{R}^{m \times r}$, $\mathbf{W} \in \mathbb{R}^{n \times r}$ are transformation matrices, $\mathbf{v}_\ell$ and $\mathbf{w}_\ell$, for $1 \le \ell \le r$, are respectively the columns of $\mathbf{V}$ and $\mathbf{W}$, and $\mathbf{g}: \mathbb{R}^r \to \mathbb{R}^r$ is as follows

$$ \mathbf{g}(t_1, \dots, t_r) = [g_1(t_1) \cdots g_r(t_r)]^{\top}, \quad (3) $$

with $g_\ell: \mathbb{R} \to \mathbb{R}$ is a univariate nonlinear function.

## B. Basis functions

The univariate nonlinear functions $g_\ell$ ($1 \le \ell \le r$) in (2) are linear combination of predefined basis functions $\{\phi_1, \dots, \phi_d\}$:

$$ g_{\ell}(t) = c_{0,\ell} + c_{1,\ell}\phi_{1}(t) + \dots + c_{d,\ell}\phi_{d}(t). \quad (4) $$

The basis functions $\phi_k$, for $1 \le k \le d$, in (4) being a priori chosen, the parameters $c_{k,\ell}$, for $0 \le k \le d$ and $1 \le \ell \le r$, are the learnable parameters.

# III. TENSOR-BASED MULTIVARIATE FUNCTION DECOMPOSITION

Representing a non-linear function **f** by a simpler representation, such as (2) has been addressed in several works [7], [9], [10] since it allows to approximate the function with a reduced number of parameters and may give insights into the understanding of the mechanisms behind the nonlinear mapping. These methods assume that the function **f** is known through a polynomial parametric representation, and were developed in the context of the decoupling problem for multivariate real polynomials. One of these methods is the Jacobian method [7] which aims at decomposing **f**(**u**) according to (2) from its first-order information. This approach relies on the observation that the Jacobian matrix of **f**, evaluated at a given point **u** is expressed as

$$ \mathbf{J}_{\mathbf{f}}(\mathbf{u}) := \begin{bmatrix} \frac{\partial f_1}{\partial u_1}(\mathbf{u}) & \cdots & \frac{\partial f_1}{\partial u_m}(\mathbf{u}) \\ \vdots & & \vdots \\ \frac{\partial f_n}{\partial u_1}(\mathbf{u}) & \cdots & \frac{\partial f_n}{\partial u_m}(\mathbf{u}) \end{bmatrix} \quad (5) $$

$$ = \mathbf{W} \cdot \text{diag}(g'_{1}(\mathbf{v}_{1}^{\top}\mathbf{u}) \cdots g'_{r}(\mathbf{v}_{r}^{\top}\mathbf{u})) \cdot \mathbf{V}^{\top}. \quad (6) $$

This result follows by applying the chain rule [7]. By stacking the Jacobian matrices evaluated at $N$ different sampling points $\mathbf{u}^{(j)} \in \mathbb{R}^m$, for $1 \le j \le N$, such that

$$ \mathcal{J}_{:,;,j} = \mathbf{J}_{\mathbf{f}}(\mathbf{u}^{(j)}), \quad (7) $$

we construct the so-called 3-order Jacobian tensor $\mathcal{J}$ of size $n \times m \times N$ following a CPD expressed as

$$ \mathcal{J} = \sum_{\ell=1}^{r} \mathbf{w}_{\ell} \otimes \mathbf{v}_{\ell} \otimes \mathbf{h}_{\ell}, \quad (8) $$

or equivalently as,

$$ \mathcal{J} = [\|\mathbf{W}, \mathbf{V}, \mathbf{H}\|]. \quad (9) $$

As shown in [7], [11], the columns of $\mathbf{H} \in \mathbb{R}^{N \times r}$ have the following structure

$$ \mathbf{h}_{\ell} = [g'_{\ell}(\mathbf{v}_{\ell}^{\top}\mathbf{u}^{(1)}), \cdots, g'_{\ell}(\mathbf{v}_{\ell}^{\top}\mathbf{u}^{(N)})]^{\top}, \quad (10) $$

which shows that the matrix factor $\mathbf{H}$ depends on $\mathbf{V}$. It is worth noting that the original approach of [7] is quite sensitive to non-uniqueness of the CPD and to noisy Jacobian tensor [12], [13], since a relaxed ALS solution is used to estimate the factors. Following [12], [13], this can be attributed to the fact that the CPD algorithm does not account for the particular structure of the Jacobian tensor (10), i.e. the relationship between the columns of the factor matrices $\mathbf{H}$ and $\mathbf{V}$. This becomes problematic especially when the uniqueness of the CPD is lost in which case, the relaxed ALS will fail to recover meaningful AFs. This motivated the development of structured CPD algorithms for the decoupling of multivariate polynomials [12], [13]. In addition, it was shown in [11] that accounting for the structure (10) leads to less restrictive uniqueness conditions compared to the unstructured case, thus enlarging the applicability range of the method.

# IV. LEARNING PROBLEM

To overcome the aforementioned drawbacks of existing methods, we propose in this section an algorithm for the structured CPD of Jacobian tensor. It consists in formulating the structured CPD as a constrained optimization problem for which a new projection strategy is introduced. Then, we propose to extend the structured CPD algorithm to a coupled matrix-tensor factorization, using jointly the first- and zeroth-order information of the data, which allows to estimate the constant terms of the approximated function.

## A. CTD-based approach

Assuming that $g_\ell(t)$ can be expressed as (4), the structured CPD of the Jacobian tensor can be stated as the following constrained optimization problem:

$$ \min_{\mathbf{W}, \mathbf{V}, \mathbf{H}} \left\| \mathcal{J} - \left[ |\mathbf{W}, \mathbf{V}, \mathbf{H}| \right] \right\|^2 \quad (11) $$

$$ \text{s.t.} \quad \mathbf{h}_{\ell} = \bar{\mathbf{X}}_{\ell} \cdot \bar{\mathbf{c}}_{\ell}, \quad \text{for } 1 \le \ell \le r $$
where $\bar{\mathbf{X}}_\ell$ is of size $N \times d$, and is given by

$$ \bar{\mathbf{X}}_{\ell} = \begin{bmatrix} \phi'_{1}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(1)}) & \cdots & \phi'_{d}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(1)}) \\ \vdots & \ddots & \vdots \\ \phi'_{1}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(N)}) & \cdots & \phi'_{d}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(N)}) \end{bmatrix}. \quad (12) $$

Function $\phi'_k(t)$, for $1 \le k \le d$, is the derivative of the univariate function $\phi_k(t)$, and the $\ell$-th coefficient vector $\bar{\mathbf{c}}_\ell$ is defined as

$$ \bar{\mathbf{c}}_{\ell} = [c_{1,\ell} \ c_{2,\ell} \ \cdots \ c_{d,\ell}]^{\top}. \quad (13) $$

The constraint in (11) directly implies that the entries of the matrix $\mathbf{H}$ follow

$$ h_{j,\ell} = g'_{\ell}(\bar{\mathbf{v}}_{\ell}^{\top} \mathbf{u}^{(j)}), \quad (14) $$

which satisfies the structural constraint in (10). The optimization problem (11) is nonlinear and nonconvex, which makes it difficult to solve.

To solve the problem (11), we propose an improved constrained alternating least squares (ALS) strategy, which is computationally more efficient than the one in [12]. The detailed description of the proposed method is given in Algorithm 1. The algorithm is composed of two parts, a first part to find an unconstrained decomposition of the tensor $\mathcal{J}$ (steps 3 to 5), then a second part to ensure that matrix $\mathbf{H}$ satisfies (14) (steps 6 to 12). Vector $\bar{\mathbf{c}}$, in Algorithm 1, is of length $rd$, and is defined as $\bar{\mathbf{c}} = [\bar{\mathbf{c}}_1^\top, \dots, \bar{\mathbf{c}}_r^\top]^\top$.

### Algorithm 1 CTD-based learning algorithm

**Input:** Tensor $\mathcal{J}$ of size $n \times m \times N$, functions $\{\phi_1, \dots, \phi_d\}$, rank $r$

**Output:** Factors $\mathbf{W}$, $\mathbf{V}$ and $\mathbf{H}$, and coefficients $\bar{\mathbf{c}}_\ell$.

1. Initialize $\mathbf{V}$, $\mathbf{H}$

2. repeat

3. Update $\mathbf{W}$ with $\min_{\mathbf{W}} \left\| \text{unfold}_1 \mathcal{J} - \mathbf{W} (\mathbf{H} \odot \mathbf{V})^\top \right\|^2$

4. Update $\mathbf{V}$ with $\min_{\mathbf{V}} \left\| \text{unfold}_2 \mathcal{J} - \mathbf{V} (\mathbf{H} \odot \mathbf{W})^\top \right\|^2$

5. Update $\mathbf{H}$ with $\min_{\mathbf{H}} \left\| \text{unfold}_3 \mathcal{J} - \mathbf{H} (\mathbf{V} \odot \mathbf{W})^\top \right\|^2$

6. for $\ell = 1, \dots, r$ do

7. Construct matrix $\bar{\mathbf{X}}_\ell$ according to (12)

8. end for

9. Compute $\bar{\mathbf{c}}$ as $\min_{\bar{\mathbf{c}}} \left\| \text{vec}(\mathbf{H}) - \begin{bmatrix} \bar{\mathbf{X}}_1 & & 0 \\ & \ddots & \\ 0 & & \bar{\mathbf{X}}_r \end{bmatrix} \cdot \bar{\mathbf{c}} \right\|^2$

10. for $\ell = 1, \dots, r$ do

11. Compute $\mathbf{h}_\ell$ such that: $\mathbf{h}_\ell = \bar{\mathbf{X}}_\ell \cdot \bar{\mathbf{c}}_\ell$

12. end for

13. until a convergence test is satisfied or maximum number of iteration is reached

14. return $\mathbf{W}$, $\mathbf{V}$, $\bar{\mathbf{c}}_\ell$.

The advantages of Algorithm 1 compared to the state-of-the-art methods can be summarized as follows. First, the proposed

solution takes into account the structure of the Jacobian tensor, as opposed to the original work [7]. Compared to the constrained algorithms in [12], [13], the proposed algorithm uses a new projection strategy. Our method constrains the factor $\mathbf{H}$ in a similar way as [12], but it involves the pseudo-inverse of a matrix of size $rN \times rd$, instead of a large matrix of size $Nm \times rd$ as in [12]. This results in highly reduced computational and storage costs.

However, the CTD-based proposed approach (Algorithm 1), like the other state-of-the-art approaches, is based on the first-order information (data derivatives) only. Therefore, it cannot estimate the constant terms of the approximated functions and introduces an estimation bias. To overcome this problem, we propose in the next section to combine the first-order information with the zeroth-order information.

### B. Constrained CMTF-based approach

1) Zeroth-order information: Let $\mathbf{F}$ be a matrix of size $n \times N$ whose rows are the evaluations of the $n$ outputs of the original subnetwork at $N$ different sampling points $\mathbf{u}^{(j)} \in \mathbb{R}^m$:

$$ \mathbf{F} = \begin{bmatrix} f_1(\mathbf{u}^{(1)}) & \cdots & f_1(\mathbf{u}^{(N)}) \\ \vdots & \ddots & \vdots \\ f_n(\mathbf{u}^{(1)}) & \cdots & f_n(\mathbf{u}^{(N)}) \end{bmatrix}. \quad (15) $$

Based on (15) and (2), we can see that $\mathbf{F}$ is subject to a rank factorization given by

$$ \mathbf{F} = \begin{bmatrix} f_1(\mathbf{u}^{(1)}) & \cdots & f_1(\mathbf{u}^{(N)}) \\ \vdots & \ddots & \vdots \\ f_n(\mathbf{u}^{(1)}) & \cdots & f_n(\mathbf{u}^{(N)}) \end{bmatrix} \quad (16) $$

$$ = \mathbf{W} \cdot \underbrace{\begin{bmatrix} g_1(\mathbf{v}_1^\top \mathbf{u}^{(1)}) & \cdots & g_1(\mathbf{v}_r^\top \mathbf{u}^{(N)}) \\ \vdots & \ddots & \vdots \\ g_r(\mathbf{v}_1^\top \mathbf{u}^{(1)}) & \cdots & g_r(\mathbf{v}_r^\top \mathbf{u}^{(N)}) \end{bmatrix}}_{\mathbf{z}^\top}, \quad (17) $$

where $\mathbf{Z}$ is of size $N \times r$. One can see from (17) that matrix $\mathbf{F}$ shares the same factor $\mathbf{W}$ with the tensor $\mathcal{J}$. Based on this observation, we build a constrained CMTF problem, involving the matrix $\mathbf{F}$ and the tensor $\mathcal{J}$.

2) Learning algorithm: The idea of the constrained CMTF-based approach is to formulate the learning of the parameters, $\mathbf{W}$, $\mathbf{V}$ and $c_{k,\ell}$, for $0 \le k \le d$ and $1 \le \ell \le r$, as follows.

$$ \min_{\mathbf{W}, \mathbf{V}, \mathbf{H}, \mathbf{Z}} \left\| \mathcal{J} - \left[ \left\| \mathbf{W}, \mathbf{V}, \mathbf{H} \right\| \right] \right\|^2 + \lambda \cdot \left\| \mathbf{F} - \mathbf{W}\mathbf{Z}^T \right\|^2 \quad (18) $$

$$ \text{s.t.} \quad h_{j,\ell} = c_{1,\ell}\phi'_{1}(\mathbf{v}_{\ell}^{\top}\mathbf{u}^{(j)}) + \cdots + c_{d,\ell}\phi'_{d}(\mathbf{v}_{\ell}^{\top}\mathbf{u}^{(j)}), $$

$$ z_{j,l} = c_{0,l} + c_{1,l}\phi_{1}(\mathbf{v}_{\ell}^{\top}\mathbf{u}^{(j)}) + \cdots + c_{d,l}\phi_{d}(\mathbf{v}_{\ell}^{\top}\mathbf{u}^{(j)}). $$

These constraints be expressed equivalently as

$$ \mathbf{h}_{\ell} = \mathbf{X}_{\ell} \cdot \mathbf{c}_{\ell}, \quad \mathbf{z}_{\ell} = \mathbf{Y}_{\ell} \cdot \mathbf{c}_{\ell}, \quad \text{for } 1 \le \ell \le r, $$
where the matrices $X_\ell$ and $Y_\ell$, of size $N \times (d+1)$, are expressed as

$$X_{\ell} = \begin{bmatrix} 0 & \phi'_{1}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(1)}) & \cdots & \phi'_{d}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(1)}) \\ \vdots & \vdots & \ddots & \vdots \\ 0 & \phi'_{1}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(N)}) & \cdots & \phi'_{d}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(N)}) \end{bmatrix}, \quad (19)$$

$$Y_{\ell} = \begin{bmatrix} 1 & \phi_{1}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(1)}) & \cdots & \phi_{d}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(1)}) \\ \vdots & \vdots & \ddots & \vdots \\ 1 & \phi_{1}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(N)}) & \cdots & \phi_{d}(\mathbf{v}_{\ell}^{\top} \mathbf{u}^{(N)}) \end{bmatrix}, \quad (20)$$

and the $\ell$-th coefficient vector $c_\ell$ is defined as

$$c_{\ell} = [c_{0,\ell} \ c_{1,\ell} \ \dots \ c_{d,\ell}]^{\top}. \quad (21)$$

To solve the problem (18), we propose an alternating least squares algorithm with constrained factors $H$ and $Z$, in which we update the coefficients of the AFs $g_1, \dots, g_r$ according to (4). The full procedure is given in Algorithm 2.

Algorithm 2 CMTF-based learning algorithm

**Input:** Tensor $\mathcal{J}$ of size $n \times m \times N$, matrix $\mathbf{F}$ of size $n \times N$, functions $\{\phi_1, \dots, \phi_d\}$, fitting parameter $\lambda$, rank $r$

**Output:** Factors $\mathbf{W}, \mathbf{V}, \mathbf{H}$ and $\mathbf{Z}$, and coefficients $c_\ell$.

1: Initialize $\mathbf{V}, \mathbf{H}, \mathbf{Z}$

2: **repeat**

3: Update $\mathbf{W}$ with $\min_{\mathbf{W}} \left\| \text{unfold}_1 \mathcal{J} - \mathbf{W} (\mathbf{H} \odot \mathbf{V})^{\top} \right\|^2 + \lambda \cdot \left\| \mathbf{F} - \mathbf{W}\mathbf{Z}^{\top} \right\|^2$

4: Update $\mathbf{V}$ with $\min_{\mathbf{V}} \left\| \text{unfold}_2 \mathcal{J} - \mathbf{V} (\mathbf{H} \odot \mathbf{W})^{\top} \right\|^2$

5: Update $\mathbf{H}$ with $\min_{\mathbf{H}} \left\| \text{unfold}_3 \mathcal{J} - \mathbf{H} (\mathbf{V} \odot \mathbf{W})^{\top} \right\|^2$

6: Update $\mathbf{Z}$ with $\min_{\mathbf{Z}} \left\| \mathbf{F} - \mathbf{W}\mathbf{Z}^{\top} \right\|^2$

7: **for** $\ell = 1 \dots r$ **do**

8: Construct matrix $\mathbf{X}_\ell$ according to (19).

9: Construct matrix $\mathbf{Y}_\ell$ according to (20).

10: **end for**

11: Compute $\mathbf{c}$ as $\min_{\mathbf{c}} \left\| \text{vec}(\mathbf{H}) - \begin{bmatrix} \mathbf{X}_1 & & 0 \\ & \ddots & \\ 0 & & \mathbf{X}_r \end{bmatrix} \cdot \mathbf{c} \right\|^2 + \lambda \cdot \left\| \text{vec}(\mathbf{Z}) - \begin{bmatrix} \mathbf{Y}_1 & & 0 \\ & \ddots & \\ 0 & & \mathbf{Y}_r \end{bmatrix} \cdot \mathbf{c} \right\|^2$

12: **for** $\ell = 1 \dots r$ **do**

13: Compute $\mathbf{h}_\ell$ such that: $\mathbf{h}_\ell = \mathbf{X}_\ell \cdot \mathbf{c}_\ell$

14: Compute $\mathbf{z}_\ell$ such that: $\mathbf{z}_\ell = \mathbf{Y}_\ell \cdot \mathbf{c}_\ell$

15: **end for**

16: **until** a convergence test is satisfied or maximum number of iteration is reached

17: **return** $\mathbf{W}, \mathbf{V}, c_\ell$.

V. PRE-TRAINED NEURAL NETWORKS COMPRESSION

To illustrate the effectiveness of tensor-based flexible neural network learning, the compression of pre-trained neural networks is considered in this section. One of the main goals of the compression is to reduce the complexity of a neural network in order to allow its implementation on embedded systems with limited computational and memory capacities.

A. Compression process

The idea here is to replace one or more consecutive layers of the original network by a single flexible NN. By providing this flexibility in the network, it is expected to increase its expressiveness, resulting in a reduced number of parameters while maintaining very good classification performance. This process is represented graphically in Fig. 2. The left hand side of the figure corresponds to a neural network with two hidden layers and fixed AFs (ReLU s in this example). The two hidden layers are replaced by the single layer with flexible AFs (on the right hand side of Fig. 2) having a lower number of neurons than the initial network, i.e. the compressed network.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Graphical representation of the compression process: (left) the original pretrained NN with fixed AFs - (right) the approximated NN with flexible AFs.</figcaption></figure>

B. Some simulation results

In this work, we consider a Convolutional NN (CNN) [14], which includes four convolutional layers using max-out nonlinear AF [15]. The network was trained to classify 24 × 24 gray-scale character images into 36 different classes, corresponding to 10 digits and 26 letters. The number of parameters in the third layer represents almost 75% of the total number of parameters of the trained CNN, with over 2 million parameters, which makes this part of the network a good target for applying a parameter reduction method. This CNN was trained on the ICDAR dataset. The images are normalized to have zero mean and unit variance. This dataset includes 185639 images in the training set and 5198 in the test set. The reported recognition accuracy [14] of the CNN is 96.97%
on the training set and 91.52% on the test set. The Normalized MSEs are defined according to

$$ \text{Tensor NMSE} = \frac{\| \hat{\mathcal{J}} - \mathcal{J} \|^2}{\| \mathcal{J} \|^2}, $$

and,

$$ \text{Matrix NMSE} = \frac{\| \hat{\mathbf{F}} - \mathbf{F} \|^2}{\| \mathbf{F} \|^2}, $$

where $\hat{\mathcal{J}}$ and $\hat{\mathbf{F}}$ refer to the estimated tensor and matrix, respectively. The Accuracy drop is defined as the difference between the classification accuracy of the original and of the approximated CNN. The Compression ratio is equal to the ratio between the number of parameters of the approximated and original layers. In the following experiment, we fix $N = 360$ and $d = 4$. The Jacobian tensor $\mathcal{J}$ is then of size $128 \times 4096 \times 360$ and the evaluation matrix $\mathbf{F}$ is of size $128 \times 360$. The decomposition basis was chosen such that $g_{\ell}(t) = c_{0,\ell} + \sum_{k=1}^{d} c_{k,\ell} \cdot \text{ReLU}(t - t_k)$. We note that CTD approach has been performed with an offset correction in this experiment.

<figure><img src="image_4.png" /><figcaption>Fig. 3. CMTF vs CTD, with $N = 360$, $d = 4$.</figcaption></figure>

Based on Fig. 3, one can note that the CMTF approach gives significantly better classification accuracy compared to the CTD, especially for higher ranks. While the CMTF approach is slightly worse than CTD in terms of the tensor NMSE, it gives much lower matrix NMSE. We can notice that, using the CMTF approach, we obtain an accuracy drop of less than 0.5%, while keeping almost 25% of the parameters of the original subnetwork in the new flexible layer. A similar behavior was observed considering other decomposition bases. This shows the interest of using the coupled approach to (i) directly estimate the constant terms of the approximated function, and (ii) to improve classification accuracy.

## VI. CONCLUSION

In this work, we have proposed a tensor-based framework for the compression of large neural networks, using flexible activation functions. The AFs are expressed as a weighted sum of predefined basis functions. By formulating the problem as a constrained coupled matrix-tensor factorization problem, we propose a new ALS-based learning algorithm, which can handle different decomposition bases. The aim of this solution

is to compress large NN, by replacing one or more layers of the initial network with a layer having flexible AFs. The learning of the new functions is performed based on a coupled factorization of a Jacobian tensor, following a constrained canonical polyadic decomposition, and a constrained evaluation matrix.

## ACKNOWLEDGMENT

This research was supported by the ANR (Agence Nationale de Recherche) grant LeaFleT (ANR-19-CE23-0021).

## REFERENCES

[1] V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky, "Speeding-up convolutional neural networks using fine-tuned CP-decomposition," CoRR, vol. abs/1412.6553, 2015.

[2] F. L. Hitchcock, "Multiple invariants and generalized rank of a p-way matrix or tensor," Journal of Mathematics and Physics, vol. 7, pp. 39-79, 1927.

[3] R. A. Harshman, "Foundations of the parafac procedure: Models and conditions for an 'explanatory' multi-modal factor analysis," UCLA Working Papers in Phonetics, vol. 16, pp. 1-84, 1970.

[4] N. Cohen, O. Sharir, and A. Shashua, "On the expressive power of deep learning: A tensor analysis," in 29th Annual Conference on Learning Theory, 2016, pp. 698-728.

[5] A. Apicella, F. Donnarumma, F. Isgrò, and R. Prevete, "A survey on modern trainable activation functions," Neural Networks, vol. 138, pp. 14-32, 2021.

[6] P. Comon, Y. Qi, and K. Usevich, "Identifiability of an X-rank decomposition of polynomial maps," SIAM Journal on Applied Algebra and Geometry, vol. 1, no. 1, pp. 388-414, 2017. [Online]. Available: https://doi.org/10.1137/16M1108388

[7] P. Dreesen, M. Ishteva, and J. Schoukens, "Decoupling multivariate polynomials using first-order information and tensor decompositions," SIAM Journal on Matrix Analysis and Applications, vol. 36, no. 12, pp. 864-879, 2015.

[8] T. G. Kolda and B. W. Bader, "Tensor decompositions and applications," SIAM Rev., vol. 51, pp. 455-500, 2009.

[9] M. Schoukens and Y. Rolain, "Cross-term elimination in parallel Wiener systems using a linear input transformation," IEEE Transactions on Instrumentation and Measurement, vol. 61, pp. 845-847, 2012.

[10] A. V. Mulders, L. Vanbeylen, and K. Usevich, "Identification of a block-structured model with several sources of nonlinearity," in European Control Conference (ECC), Strasbourg, France, 2014.

[11] K. Usevich, P. Dreesen, and M. Ishteva, "Decoupling multivariate polynomials: Interconnections between tensorizations," Journal of Computational and Applied Mathematics, vol. 363, pp. 22-34, 2020.

[12] G. Hollander, "Multivariate polynomial decoupling in nonlinear system identification," Ph.D. dissertation, Vrije Universiteit Brussel, 2018.

[13] Y. Zniyed, K. Usevich, S. Miron, and D. Brie, "Learning nonlinearities in the decoupling problem with structured CPD," in 16th IFAC Symposium on System Identification, Padova, Italy, 2021.

[14] M. Jaderberg, A. Vedaldi, and A. Zisserman, "Deep features for text spotting," in Computer Vision - ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, pp. 512-528.

[15] I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, "Maxout networks," in Proceedings of the 30th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, S. Dasgupta and D. McAllester, Eds., vol. 28, no. 3. Atlanta, Georgia, USA: PMLR, 17-19 Jun 2013, pp. 1319-1327. [Online]. Available: http://proceedings.mlr.press/v28/goodfellow13.html