<figure><img src="image_1.png" /></figure>

HAL
open science

# Compressing Neural Networks with Two-Layer Decoupling

Joppe de Jonghe, Konstantin Usevich, Philippe Dreesen, Mariya Ishteva

► To cite this version:

Joppe de Jonghe, Konstantin Usevich, Philippe Dreesen, Mariya Ishteva. Compressing Neural Networks with Two-Layer Decoupling. 2023 IEEE 9th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), Dec 2023, Los Sueños, Costa Rica. pp.226-230, 10.1109/CAMSAP58249.2023.10403509. hal-05008394

HAL Id: hal-05008394

https://hal.science/hal-05008394v1

Submitted on 27 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Compressing neural networks with two-layer decoupling

Joppe De Jonghe

Dept. of computer science

KU Leuven

Geel, Belgium

joppe.dejonghe@kuleuven.be

Konstantin Usevich

CNRS

Université de Lorraine

Nancy, France

konstantin.usevich@cnrs.fr

Philippe Dreesen

DACS

Maastricht University

Maastricht, Netherlands

philippe.dreesen@gmail.com

Mariya Ishteva

Dept. of computer science

KU Leuven

Geel, Belgium

mariya.ishteva@kuleuven.be

**Abstract**—The single-layer decoupling problem has recently been used for the compression of neural networks. However, methods that are based on the single-layer decoupling problem only allow the compression of a neural network to a single flexible layer. As a result, compressing more complex networks leads to worse approximations of the original network due to only having one flexible layer. Having the ability to compress to more than one flexible layer thus allows to better approximate the underlying network compared to compression into only a single flexible layer. Performing compression into more than one flexible layer corresponds to solving a multi-layer decoupling problem. As a first step towards general multi-layer decoupling, this work introduces a method for solving the two-layer decoupling problem in the approximate case. This method enables the compression of neural networks into two flexible layers.

**Index Terms**—tensor, tensor decomposition, decoupling, compression, neural network

## I. INTRODUCTION

Neural networks continue to grow in size annually, leading to increased memory requirements, computational power, and time needed for computation. This poses challenges for real-time applications and deployment on Internet of Things devices [16], [17]. Consequently, significant research efforts have focused on compressing and accelerating neural networks such as low-rank factorization methods, filter and weight pruning and knowledge distillation [2], [5], [9], [14]. A more recent approach involves leveraging the single-layer decoupling problem [7] and its neural network interpretation [18].

Zniyed et al. [18] used the single-layer decoupling problem to compress neural networks by interpreting a trained (sub)network as a multivariate function $f(x)$. By then approximating a solution to the single-layer decoupling problem, the original network is approximated by a network consisting of a single hidden layer with $r$ neurons where the internal functions $g_i$ of the decoupled representation form the activation functions of the neurons. These so-called flexible activation functions $g_i$ [18] differ from standard activation functions such as the rectified linear unit [8] in the sense that they have a flexible shape, to best approximate $f(x)$.

A downside of this method is that currently there is only the possibility to compress a network to a single layer of flexible activation functions, regardless of the function complexity. Although it follows from the universal approximation theorem

that any continuous function on a compact set can be adequately approximated by a neural network with a single hidden layer [3], this is not useful when the goal is compression.

Decoupling to multiple layers forms a solution to gain better approximations of more complex function $f(x)$, while still maintaining good compression. The first step in this direction is approximating solutions to the two-layer decoupling problem, which is defined in this work. This idea forms a natural extension of the single-layer decoupling problem.

## II. NOTATIONS AND DEFINITIONS

Matrices and vectors are denoted with bold capital and lowercase letters, respectively. Tensors are denoted by caligraphic capital letters. For a third order tensor $\mathcal{X}$ of size $I \times J \times K$, the $i$-th horizontal, $j$-th lateral and $k$-th frontal slice are denoted by $\mathcal{X}_{i,;,;}$, $\mathcal{X}_{:,j,;}$ and $\mathcal{X}_{:,;,k}$ respectively. The operation unfold$_k(\mathcal{X})$ unfolds the tensor $\mathcal{X}$ over its $k$-th mode as described in [11]. The $i$-th row and $j$-th column of a matrix $\mathbf{A}$ are denoted as $\mathbf{A}^{i;}$ and $\mathbf{A}^{;,j}$ respectively. The outer, Kronecker and Khatri-Rao product are denoted by $\circ$, $\otimes$, $\odot$, respectively. Finally, the vec(.) operation vectorizes the given matrix or tensor as described in [11] and $\|\cdot\|$ denotes the norm of a tensor, defined as the square root of the sum of the squares of its elements.

### A. The PARATUCK2 decomposition

The PARATUCK2 (PT2) decomposition [11] of a third-order tensor $\mathcal{X} \in \mathbb{R}^{I \times J \times K}$ is a decomposition where the frontal slices of $\mathcal{X}$ take on the form

$$ \mathcal{X}_{:,;,k} = \mathbf{A} \mathbf{D}_{A}^{(k)} \mathbf{B} \mathbf{D}_{C}^{(k)} \mathbf{C}^{T}, \text{ for } k = 1, 2, \dots, K, \quad (1) $$

where $\mathbf{A} \in \mathbb{R}^{I \times r_2}$, $\mathbf{B} \in \mathbb{R}^{r_2 \times r_1}$, $\mathbf{C} \in \mathbb{R}^{J \times r_1}$ and $\mathbf{D}_A^{(k)} \in \mathbb{R}^{r_2 \times r_2}$ and $\mathbf{D}_C^{(k)} \in \mathbb{R}^{r_1 \times r_1}$ diagonal matrices. Let the column vectors $\mathbf{d}_k^A$ and $\mathbf{d}_k^C$ denote the diagonals of $\mathbf{D}_A^{(k)}$ and $\mathbf{D}_C^{(k)}$, then the decomposition is characterized by a pair of ranks $(r_2, r_1)$ and five factor matrices $\mathbf{A}, \mathbf{B}, \mathbf{C}, \mathbf{D}_A = [\mathbf{d}_1^A \ \mathbf{d}_2^A \ \dots \ \mathbf{d}_K^A]^T$ and $\mathbf{D}_C = [\mathbf{d}_1^C \ \mathbf{d}_2^C \ \dots \ \mathbf{d}_K^C]^T$.

### B. Slice-wise scaling ambiguities and fitting the model

A PT2 decomposition of a third-order tensor is subject to scaling and permutation ambiguities. The scaling ambiguities that are problematic for the two-layer decoupling problem
occur along the frontal slices. More specifically, Harshman and Lundy [10] have shown that if a third-order tensor $\mathcal{X}$ has a PT2 decomposition according to (1), then there exist equivalent decompositions

$$ \mathcal{X}_{:,;,k} = \mathbf{A} \hat{\mathbf{D}}_{\mathbf{A}}^{(k)} \mathbf{B} \hat{\mathbf{D}}_{\mathbf{C}}^{(k)} \mathbf{C}^{\mathrm{T}}, \text{ for } k = 1, 2, \dots, K, $$

where $\hat{\mathbf{D}}_{\mathbf{A}}^{(k)}$ and $\hat{\mathbf{D}}_{\mathbf{C}}^{(k)}$ are scaled with a non-zero scalar $\alpha_k$ as $\hat{\mathbf{D}}_{\mathbf{A}}^{(k)} = \alpha_k \cdot \mathbf{D}_{\mathbf{A}}^{(k)}$ and $\hat{\mathbf{D}}_{\mathbf{C}}^{(k)} = \alpha_k^{-1} \cdot \mathbf{D}_{\mathbf{C}}^{(k)}$. These scaling factors can be different per frontal slice and require extra care when managing the decomposition ambiguities [15].

Besides this, the PT2 decomposition is known to be difficult to fit [10]. As a result, many applications of the PT2 decomposition simplify the problem by incorporating that certain factor matrices are known [1], [4], [13], [15]. In this work, no factor matrices are assumed to be known.

### III. THE TWO-LAYER DECOUPLING PROBLEM

This work builds upon the single-layer decoupling problem introduced in [7] by extending it to a two-layer decoupling setting, similar to [15]. The two-layer decoupling problem (Fig. 1) is characterized by the decoupled representation

$$ \mathbf{f}(\mathbf{x}) = \mathbf{W}\mathbf{g}(\mathbf{V}^{\mathrm{T}}\mathbf{h}(\mathbf{Z}^{\mathrm{T}}\mathbf{x})), \quad (2) $$

of a multivariate vector function $\mathbf{f}: \mathbb{R}^m \to \mathbb{R}^n$ with input $\mathbf{x} \in \mathbb{R}^m$, where $\mathbf{W} \in \mathbb{R}^{n \times r_2}$, $\mathbf{V} \in \mathbb{R}^{r_1 \times r_2}$, $\mathbf{Z} \in \mathbb{R}^{m \times r_1}$, $\mathbf{g}: \mathbb{R}^{r_2} \to \mathbb{R}^{r_2}$ and $\mathbf{h}: \mathbb{R}^{r_1} \to \mathbb{R}^{r_1}$. Here, $\mathbf{g}$ and $\mathbf{h}$ are comprised of univariate functions $g_i: \mathbb{R} \to \mathbb{R}$, $h_j: \mathbb{R} \to \mathbb{R}$, for $i=1,2,\dots,r_2$ and $j=1,2,\dots,r_1$. More specifically, let $\mathbf{a} \in \mathbb{R}^{r_2}$ and $\mathbf{b} \in \mathbb{R}^{r_1}$, then

$$ \mathbf{g}(\mathbf{a}) = [g_1(a_1) \ g_2(a_2) \ \dots \ g_{r_2}(a_{r_2})]^T, $$

$$ \mathbf{h}(\mathbf{b}) = [h_1(b_1) \ h_2(b_2) \ \dots \ h_{r_1}(b_{r_1})]^T. $$

Solving the two-layer decoupling problem thus means determining the matrices $\mathbf{W}$, $\mathbf{V}$, $\mathbf{Z}$ and coefficients of the internal function $g_i$ and $h_j$ such that (2) holds.

#### A. Representation of internal functions

This work represents the internal functions $g_i$ and $h_j$ as polynomials of degree $d_g$ and $d_h$, respectively. Specifically, if $\mathbf{a} \in \mathbb{R}$ and $\mathbf{b} \in \mathbb{R}$, then the representation is

$$ \begin{aligned} g_i(\mathbf{a}) &= c_{g_i,0} + c_{g_i,1} \cdot \mathbf{a} + c_{g_i,2} \cdot \mathbf{a}^2 + \dots + c_{g_i,d_g} \cdot \mathbf{a}^{d_g}, \\ h_j(\mathbf{b}) &= c_{h_j,0} + c_{h_j,1} \cdot \mathbf{b} + c_{h_j,2} \cdot \mathbf{b}^2 + \dots + c_{h_j,d_h} \cdot \mathbf{b}^{d_h}, \end{aligned} $$

where the coefficients $c_{i,j}$ need to be found or learned as part of a solution to the two-layer decoupling problem.

#### B. Solution strategy

In order to solve the two-layer decoupling problem, a similar method is followed as for the single-layer decoupling problem in [7]. Namely, the first-order information, captured by the Jacobian matrix $\mathbf{J}(\mathbf{x})$ of $\mathbf{f}(\mathbf{x})$ is used, where $\mathbf{J}(\mathbf{x})$ is given by

$$ \mathbf{J}(\mathbf{x}) = \begin{bmatrix} \frac{\partial f_1}{\partial x_1}(\mathbf{x}) & \cdots & \frac{\partial f_1}{\partial x_m}(\mathbf{x}) \\ \vdots & \ddots & \vdots \\ \frac{\partial f_n}{\partial x_1}(\mathbf{x}) & \cdots & \frac{\partial f_n}{\partial x_m}(\mathbf{x}) \end{bmatrix} \in \mathbb{R}^{n \times m}. $$

<figure><img src="image_3.png" /><figcaption>Fig. 1. The two-layer decoupling problem. Given $\mathbf{f}(\mathbf{x})$, determine the matrices $\mathbf{W}$, $\mathbf{V}$, $\mathbf{Z}$ and the coefficients of the internal functions $g_i$ and $h_j$.</figcaption></figure>

This Jacobian matrix is evaluated in $S$ sampling points $\mathbf{x}^{(1)}, \dots, \mathbf{x}^{(S)}$, and the resulting $\mathbf{J}(\mathbf{x}^{(1)}), \dots, \mathbf{J}(\mathbf{x}^{(S)})$ are stacked into the third-order tensor $\mathcal{J} \in \mathbb{R}^{n \times m \times S}$. Applying the chain rule to (2) gives

$$ \begin{aligned} \mathbf{J}(\mathbf{x}^{(s)}) &= \mathbf{W} \operatorname{diag}(g_i'(\mathbf{v}_i^T \mathbf{h}(\mathbf{b}^{(s)}))) \mathbf{V}^T \operatorname{diag}(h_j'(\mathbf{z}_j^T \mathbf{x}^{(s)})) \mathbf{Z}^T \\ &= \mathbf{W} \mathbf{D}_g^{(s)} \mathbf{V}^T \mathbf{D}_h^{(s)} \mathbf{Z}^T, \end{aligned} $$

where $\mathbf{b}^{(s)} = \mathbf{Z}^T \mathbf{x}^{(s)}$ [15]. As a result, the frontal slices of the tensor $\mathcal{J}$ have the structure

$$ \mathcal{J}_{:,;,s} = \mathbf{W} \mathbf{D}_g^{(s)} \mathbf{V}^T \mathbf{D}_h^{(s)} \mathbf{Z}^T, \text{ for } s = 1, 2, \dots, S, $$

by construction. Thus, the structure of tensor $\mathcal{J}$ admits a PT2 decomposition according to (1). This PT2 decomposition of $\mathcal{J}$ yields the matrices $\mathbf{W}$, $\mathbf{V}$ and $\mathbf{Z}$ as well as first-order information of the internal functions $g_i$ and $h_j$, for $i=1, \dots, r_2$ and $j=1, \dots, r_1$. This first-order information is captured in factor matrices $\mathbf{D}_g \in \mathbb{R}^{S \times r_2}$ and $\mathbf{D}_h \in \mathbb{R}^{S \times r_1}$ of the PT2 decomposition where $\mathbf{D}_g^{s,;}$ is equal to the diagonal of $\mathbf{D}_g^{(s)}$ and the same holds for $\mathbf{D}_h^{s,;}$ and $\mathbf{D}_h^{(s)}$, for $s=1, 2, \dots, S$.

The solution strategy can be summarized as follows:

1. Evaluate the Jacobian $\mathbf{J}(\mathbf{x})$ in $S$ points $\mathbf{x}^{(1)}, \dots, \mathbf{x}^{(S)}$.

2. Stack the Jacobians $\mathbf{J}(\mathbf{x}^{(1)}), \dots, \mathbf{J}(\mathbf{x}^{(S)})$ into tensor $\mathcal{J}$.

3. Compute a PT2 decomposition of $\mathcal{J}$, yielding $\mathbf{W}$, $\mathbf{V}$ and $\mathbf{Z}$ as well as first order information of the internal functions in the factor matrices $\mathbf{D}_g$ and $\mathbf{D}_h$.

4. Use the first order information captured in $\mathbf{D}_g$ and $\mathbf{D}_h$ to determine the coefficients of the internal functions.

### IV. FORMULATING THE TWO-LAYER DECOUPLING PROBLEM AS AN OPTIMIZATION PROBLEM

This work formulates the two-layer decoupling problem as an optimization problem. This section describes the optimization problem and its construction and the next section describes an algorithm to solve it.

#### A. Construction of optimization problem

Because the PT2 decomposition of $\mathcal{J}$ is crucial to the proposed solution strategy, we first discuss the basis of the optimization problem which is an unstructured PT2 decomposition of $\mathcal{J}$. This unstructured decomposition is given by

$$ \min_{\mathbf{W}, \mathbf{D}_{g_i}, \mathbf{V}, \mathbf{D}_h, \mathbf{Z}} \sum_{s=1}^{S} \| \mathcal{J}_{:,;,s} - \mathbf{W} \mathbf{D}_g^{(s)} \mathbf{V}^T \mathbf{D}_h^{(s)} \mathbf{Z}^T \|^2. \quad (3) $$

In addition, we need to consider the following two problems:
1) Equation (3) does not guarantee that the computed $D_g$ and $D_h$ have the required structure, i.e., contain the first-order information of the internal functions $g_i$ and $h_j$.

2) Equation (3) does not take into account the slice-wise scaling ambiguities mentioned in Section II-B.

In order to solve these problems, constraints on the columns of $D_g$ and $D_h$ are added to optimization problem (3). More specifically, the structure of $D_g$ requires that

$$ 
\begin{aligned}
\mathbf{D}_g^{i} &= [g'_i(a_i^{(1)}) g'_i(a_i^{(2)}) \cdots g'_i(a_i^{(S)})]^T \\
&= \begin{bmatrix} 0 & 1 & 2a_i^{(1)} & \cdots & d_g(a_i^{(1)})^{d_g-1} \\ \vdots & \vdots & \vdots & & \vdots \\ 0 & 1 & 2a_i^{(S)} & \cdots & d_g(a_i^{(S)})^{d_g-1} \end{bmatrix} \begin{bmatrix} c_{g_i,0} \\ \vdots \\ c_{g_i,d_g} \end{bmatrix} \\
&= \mathbf{X}_{g_i} \cdot \mathbf{c}_{g_i},
\end{aligned}
$$

where $\mathbf{X}_{g_i} \in \mathbb{R}^{S \times d_g+1}$ and $\mathbf{c}_{g_i} \in \mathbb{R}^{d_g+1}$, for $i=1,2,\dots,r_2$ and $a_i^{(s)} = \mathbf{v}_i^T \mathbf{h}(\mathbf{Z}^T \mathbf{x}^{(s)})$, for $s=1,2,\dots,S$. To enforce this structure, the constraints $\mathbf{D}_g^{i,j} = \mathbf{X}_{g_i} \mathbf{c}_{g_i}$ are added to optimization problem (3). The same reasoning can be made for the columns of $D_h$ which leads to similar constraints $\mathbf{D}_h^{i,j} = \mathbf{X}_{h_j} \mathbf{c}_{h_j}$, for $j=1,2,\dots,r_1$, that are added in the optimization problem where $\mathbf{X}_{h_j} \in \mathbb{R}^{S \times d_h+1}$ and $\mathbf{c}_{h_j} \in \mathbb{R}^{d_h+1}$. The resulting optimization problem is given by

$$ 
\begin{aligned}
\min_{\mathbf{W}, \mathbf{D}_g, \mathbf{V}, \mathbf{D}_h, \mathbf{Z}} \sum_{s=1}^{S} \| \mathcal{J}_{:,s} - \mathbf{W} \mathbf{D}_g^{(s)} \mathbf{V}^T \mathbf{D}_h^{(s)} \mathbf{Z}^T \|^2 \\
\text{s.t.} \quad \mathbf{D}_g^{i} = \mathbf{X}_{g_i} \mathbf{c}_{g_i} \quad \text{for } i=1,2,\dots,r_2, \\
\mathbf{D}_h^{j} = \mathbf{X}_{h_j} \mathbf{c}_{h_j} \quad \text{for } j=1,2,\dots,r_1.
\end{aligned}
\quad (4) $$

Because these constraints work on the columns of $D_g$ and $D_h$, they also solve the slice-wise scaling ambiguities as they enforce structure across the frontal slices of $\mathcal{J}$.

A final problem to be solved is that optimization problem (4) is unable to estimate the constant terms of the internal functions $g_i$ and $h_j$ because it only uses first-order information of $\mathbf{f}(\mathbf{x})$. This causes a bias in the computed decoupling that depends on the start values of the constant terms of the $g_i$ and $h_j$ functions. We refer to this problem as the *bias problem*.

## B. Solving the bias problem and final optimization problem

This solution solves the bias problem by integrating zeroth-order information of $\mathbf{f}(\mathbf{x})$ in optimization problem (4). The matrix $\mathbf{F} \in \mathbb{R}^{n \times S}$ encapsulates this zeroth-order information

$$ 
\begin{aligned}
\mathbf{F} &= [\mathbf{f}(\mathbf{x}^{(1)}) \ \mathbf{f}(\mathbf{x}^{(2)}) \ \cdots \ \mathbf{f}(\mathbf{x}^{(S)})] \\
&= \mathbf{W} \cdot [\mathbf{g}(\mathbf{a}^{(1)}) \ \mathbf{g}(\mathbf{a}^{(2)}) \ \cdots \ \mathbf{g}(\mathbf{a}^{(S)})] \\
&= \mathbf{W} \cdot \mathbf{R}^T
\end{aligned}
\quad (5) $$

where $\mathbf{R} \in \mathbb{R}^{S \times r_2}$ and $\mathbf{a}^{(s)} = \mathbf{V}^T \mathbf{h}(\mathbf{Z}^T \mathbf{x}^{(s)})$, for $s=1,2,\dots,S$. The fact that $\mathbf{F} = \mathbf{W} \mathbf{R}^T$ shows that the factorization of the matrix $\mathbf{F}$ and the decomposition of the tensor $\mathcal{J}$ are coupled through the factor matrix $\mathbf{W}$. Because of this,

the resulting final optimization problem is a *coupled matrix-factorization problem* (CMTF) [12]

$$ 
\begin{aligned}
\min_{\mathbf{W}, \mathbf{D}_g, \mathbf{V}, \mathbf{D}_h, \mathbf{Z}, \mathbf{R}} \sum_{s=1}^{S} \| \mathcal{J}_{:,s} - \mathbf{W} \mathbf{D}_g^{(s)} \mathbf{V}^T \mathbf{D}_h^{(s)} \mathbf{Z}^T \|^2 \\
+ \lambda \| \mathbf{F} - \mathbf{W} \mathbf{R}^T \|^2
\end{aligned}
$$

$$ 
\begin{aligned}
\text{s.t.} \quad \mathbf{D}_g^{i,j} &= \mathbf{X}_{g_i} \mathbf{c}_{g_i} \quad \text{for } i=1,2,\dots,r_2 \\
\mathbf{D}_h^{i,j} &= \mathbf{X}_{h_j} \mathbf{c}_{h_j} \quad \text{for } j=1,2,\dots,r_1 \\
\mathbf{R}^{i,j} &= \mathbf{Y}_{g_i} \mathbf{c}_{g_i} \quad \text{for } i=1,2,\dots,r_2,
\end{aligned}
$$

where $\lambda$ is an adaptable parameter that determines how much weight is given to the matrix factorization. The integration of the zeroth-order information $\mathbf{F}$ allows to directly estimate the constant terms of the internal functions $g_i$, the constant terms of the functions $h_j$ stay equal to their start value. There is also an additional constraint on the columns of $\mathbf{R}$ in order to enforce its structure given in (5). In this constraint the matrix $\mathbf{Y}_{g_i} \in \mathbb{R}^{S \times d_g+1}$ (for $i=1,2,\dots,r_2$) is defined as

$$ 
\mathbf{Y}_{g_i} = \begin{bmatrix} 1 & a_i^{(1)} & (a_i^{(1)})^2 & \cdots & (a_i^{(1)})^{d_g} \\ \vdots & \vdots & \vdots & & \vdots \\ 1 & a_i^{(S)} & (a_i^{(S)})^2 & \cdots & (a_i^{(S)})^{d_g} \end{bmatrix}.
$$

## V. THE CMTF-PT2 ALGORITHM

Finally, the optimization problem is solved by means of an *alternating least squares* (ALS) algorithm that uses a projection strategy to satisfy the constraints. ALS for fitting the PT2 model is known to have convergence issues [10]. Despite this, an ALS approach is chosen due to its simplicity and because neural network compression does not require a unique decomposition of the tensor $\mathcal{J}$. Algorithm 1 describes the full algorithm, called CMTF-PT2.

The updates of Algorithm 1 use matrices with specific constructions, namely, the matrices $\mathbf{F}_{Z^T}$, $\mathbf{F}_{D_h^{(s)}}$, $\mathbf{F}_{V^T}$, $\mathbf{F}_{D_g^{(s)}}$ and $\mathbf{F}_{W}$. These matrices have the following structure

$$ 
\begin{aligned}
\mathbf{F}_{Z^T} &= \begin{bmatrix} \mathbf{W} \cdot \mathbf{D}_g^{(1)} \cdot \mathbf{V}^T \cdot \mathbf{D}_h^{(1)} \\ \vdots \\ \mathbf{W} \cdot \mathbf{D}_g^{(S)} \cdot \mathbf{V}^T \cdot \mathbf{D}_h^{(S)} \end{bmatrix} \in \mathbb{R}^{n S \times r_1}, \\
\mathbf{F}_{V^T} &= \begin{bmatrix} (\mathbf{Z} \cdot \mathbf{D}_h^{(1)}) \otimes (\mathbf{W} \cdot \mathbf{D}_g^{(1)}) \\ \vdots \\ (\mathbf{Z} \cdot \mathbf{D}_h^{(S)}) \otimes (\mathbf{W} \cdot \mathbf{D}_g^{(S)}) \end{bmatrix} \in \mathbb{R}^{n m S \times r_1 r_2}, \\
\mathbf{F}_{W} &= \begin{bmatrix} (\mathbf{D}_g^{(1)} \cdot \mathbf{V}^T \cdot \mathbf{D}_h^{(1)} \cdot \mathbf{Z}^T)^T \\ \vdots \\ (\mathbf{D}_g^{(S)} \cdot \mathbf{V}^T \cdot \mathbf{D}_h^{(S)} \cdot \mathbf{Z}^T)^T \end{bmatrix}^T \in \mathbb{R}^{r_2 \times m S}, \\
\mathbf{F}_{D_g^{(s)}} &= (\mathbf{V}^T \cdot \mathbf{D}_h^{(s)} \cdot \mathbf{Z}^T)^T \in \mathbb{R}^{m \times r_2}, \\
\mathbf{F}_{D_h^{(s)}} &= \mathbf{W} \cdot \mathbf{D}_g^{(s)} \cdot \mathbf{V}^T \in \mathbb{R}^{n \times r_1},
\end{aligned}
$$

for $s=1,2,\dots,S$. Bro [1] describes the derivations of these matrices in more detail. Besides this, the matrices
$\mathbf{X} \in \mathbb{R}^{r_2 S \times r_2 (d_g + 1)}$ and $\mathbf{Y} \in \mathbb{R}^{r_2 S \times r_2 (d_g + 1)}$ on line 15 of Algorithm 1 are block diagonal matrices with diagonal elements $\mathbf{X}_{g_1}, \dots, \mathbf{X}_{g_{r_2}}$ and $\mathbf{Y}_{g_1}, \dots, \mathbf{Y}_{g_{r_2}}$ respectively.

**Algorithm 1 CMTF-PT2 algorithm**

Require: $\mathcal{J}$, $\mathbf{F}$, $r_1$, $r_2$, $\lambda$

1: Initialize $\mathbf{W}$, $\mathbf{D}_g$, $\mathbf{V}^T$, and $\mathbf{D}_h$, $\mathbf{c}_{\mathbf{D}_g}$ and $\mathbf{c}_{\mathbf{D}_h}$

2: **while** stopping criteria not met **do**

3: Update $\mathbf{Z}^T$ with $\min_{\mathbf{Z}^T} \| \text{unfold}_2(\mathcal{J})^T - \mathbf{F}_{\mathbf{Z}^T} \mathbf{Z}^T \|^2$

4: Update $\mathbf{D}_h$ with

$$ \min_{\mathbf{D}_h^{:,i}} \| \text{vec}(\mathcal{J}_{:,i,s}) - (\mathbf{Z} \odot \mathbf{F}_{\mathbf{D}_h^{(s)}}) \mathbf{D}_h^{s:,T} \|^2, $$
for $s = 1, 2, \dots, S$

5:

6: Construct $\mathbf{X}_{h_j}$, for $j = 1, 2, \dots, r_1$

7: Compute $\mathbf{c}_{\mathbf{D}_h}$ from

$$ \min_{\mathbf{c}_{\mathbf{D}_h}} \| \text{vec}(\mathbf{D}_h) - \begin{bmatrix} \mathbf{X}_{h_1} & 0 \\ \vdots & \vdots \\ 0 & \mathbf{X}_{h_{r_1}} \end{bmatrix} \mathbf{c}_{\mathbf{D}_h} \|^2 $$

8: Set $\mathbf{D}_h^{:,j} = \mathbf{X}_{h_j} \mathbf{c}_{h_j}$, for $j = 1, 2, \dots, r_1$

9:

10: Update $\mathbf{V}^T$ with $\min_{\mathbf{V}^T} \| \text{vec}(\mathcal{J}) - \mathbf{F}_{\mathbf{V}^T} \text{vec}(\mathbf{V}^T) \|^2$

11: Update $\mathbf{D}_g$ with

$$ \min_{\mathbf{D}_g^{:,i}} \| \text{vec}(\mathcal{J}_{:,i,s}) - (\mathbf{F}_{\mathbf{D}_g^{(s)}} \odot \mathbf{W}) \mathbf{D}_g^{s:,T} \|^2, $$
for $s = 1, 2, \dots, S$

12: Update $\mathbf{R}$ with $\min_{\mathbf{R}} \| \mathbf{F} - \mathbf{W} \mathbf{R}^T \|^2$

13:

14: Construct $\mathbf{X}_{g_i}$ and $\mathbf{Y}_{g_i}$, for $i = 1, 2, \dots, r_2$

15: Compute $\mathbf{c}_{\mathbf{D}_g}$ from

$$ \min_{\mathbf{c}_{\mathbf{D}_g}} \| \text{vec}(\mathbf{D}_g) - \mathbf{X} \mathbf{c}_{\mathbf{D}_g} \|^2 + \lambda \| \text{vec}(\mathbf{R}) - \mathbf{Y} \mathbf{c}_{\mathbf{D}_g} \|^2 $$

16: Set $\mathbf{D}_g^{:,i} = \mathbf{X}_{g_i} \mathbf{c}_{g_i}$, for $i = 1, 2, \dots, r_2$

17: Set $\mathbf{R}^{:,i} = \mathbf{Y}_{g_i} \mathbf{c}_{g_i}$, for $i = 1, 2, \dots, r_2$

18:

19: Update $\mathbf{W}$ with

$$ \min_{\mathbf{W}} \| \text{unfold}_1(\mathcal{J}) - \mathbf{W} \mathbf{F}_{\mathbf{W}} \|^2 + \lambda \| \mathbf{F} - \mathbf{W} \mathbf{R}^T \|^2 $$

20: **end while**

21: **Result** $\mathbf{W}$, $\mathbf{D}_g$, $\mathbf{V}^T$, $\mathbf{D}_h$, $\mathbf{Z}^T$, $\mathbf{R}^T$, $\mathbf{c}_{\mathbf{D}_h}$, $\mathbf{c}_{\mathbf{D}_g}$

## VI. RESULTS COMPRESSING A PRETRAINED NETWORK

The experiment in this section compresses a simple fully connected neural network, pretrained on the MNIST dataset [6], with single-layer and two-layer decoupling. The network consists of an input layer of size 784, followed by three hidden layers and an output layer of sizes 80, 60, 40 and 10 respectively. These layers use the exponential linear unit [8] activation function. We compressed the hidden layers of size 60 and 40, with Algorithm 1, into two flexible layers consisting of $r_1$ and $r_2$ neurons respectively where $r_1$ is fixed to 20 and $r_2$ is varied (Fig. 2). Besides this, the CMTF algorithm from [18] was used to generate the results for compression to a single flexible layer with $r$ neurons. For comparison reasons, the number of neurons in the single and two-layer compression are chosen such that vertically aligned

<figure><img src="image_4.png" /><figcaption>Fig. 2. The accuracy drop (%), tensor NMSE and matrix NMSE values for two-layer decoupling (blue line, square) with $r_1 = 20$, $r_2 \in \{5, 7, 10, 12, 15, 17, 20\}$, $S = 200$ and for one-layer decoupling (orange line, diamond) with $r \in \{19, 20, 21, 22, 23, 24, 25\}$ (activation functions of degree 4). Vertically aligned results yield similar compression ratios to allow a fair comparison.</figcaption></figure>

results in Fig. 2 yield similar compression ratios. In addition, the number of sampling points $S$ is set to 200 (20 random samples per MNIST class) and $d_g = d_h = 4$. No specific scheme is used to determine the number of samples since the CMTF-PT2 method is not designed to take into account the uniqueness of the tensor $\mathcal{J}$. The tensor and matrix NMSE are

$$ \text{Tensor NMSE} = \frac{\|\mathcal{J} - \hat{\mathcal{J}}\|^2}{\|\mathcal{J}\|^2}, \quad \text{Matrix NMSE} = \frac{\|\mathbf{F} - \hat{\mathbf{F}}\|^2}{\|\mathbf{F}\|^2}, $$

where $\hat{\mathcal{J}}$ and $\hat{\mathbf{F}}$ denote the approximations of $\mathcal{J}$ and $\mathbf{F}$ found by the algorithm. The Accuracy drop is the absolute drop in accuracy, denoted as a percentage, of the network's accuracy on the test data after compression. On the test data, the non-compressed network has an accuracy of 97.17%.

Fig. 2 shows that when $r_2 \in \{10, 12, 15, 17, 20\}$, the two layer decoupling reaches accuracy drops below 2.54%, the lowest accuracy drop reached with single-layer decoupling. The best accuracy drop is 1.69%, reached by two-layer decoupling, for $r_1 = r_2 = 20$. The best tensor and matrix NMSE of 0.44 and 0.1769 are reached by the single and two-layer decoupling respectively, for $r = 24$ and $r_2 = 20$.

## VII. CONCLUSION AND FUTURE WORK

This work introduced a CMTF method for the two-layer decoupling problem. The approach has been used to compress a simple neural network pre-trained on MNIST, showing the two-layer decoupling results in a significant improvement.

Future work includes optimizing or adapting the current CMTF-PT2 algorithm to be able to work with a tensor $\mathcal{J}$ of large dimensions. Additionally, an analysis of the convergence properties of Algorithm 1 needs to be done since the used projection strategy influences the convergence behaviour of the ALS algorithm. More general future work pertains to generalizing the one and two-layer decoupling problems to a generalized $n$-layer decoupling problem.

## ACKNOWLEDGMENT

This research was supported by KU Leuven start-up grant STG/19/036 and C2 project 3E230536, and ANR grant LeaFleT (ANR-19-CE23-0021).
REFERENCES

[1] Rasmus Bro. Multi-way analysis in the food industry, model, algorithms and applications. Doctoral Thesis, University of Amsterdam, 1998.

[2] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35(1):126-136, 2018.

[3] Balázs Csanád Csáji et al. Approximation with artificial neural networks. Faculty of Sciences, Etvs Lornd University, Hungary, 24(48):7, 2001.

[4] Pedro Marinho R de Oliveira, C Alexandre Rolim Fernandes, Gérard Favier, and Rémy Boyer. PARATUCK semi-blind receivers for relaying multi-hop MIMO systems. Digital Signal Processing, 92:127-138, 2019.

[5] Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. *Proceedings of the IEEE*, 108(4):485-532, 2020.

[6] Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. *IEEE signal processing magazine*, 29(6):141-142, 2012.

[7] Philippe Dreesen, Mariya Ishteva, and Johan Schoukens. Decoupling multivariate polynomials using first-order information and tensor de-compositions. *SIAM Journal on Matrix analysis and Applications*, 36(2):864-879, 2015.

[8] Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Baran Chaudhuri. Activation functions in deep learning: A comprehensive survey and benchmark. *Neurocomputing*, 2022.

[9] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. *International Journal of Computer Vision*, 129:1789-1819, 2021.

[10] Richard A Harshman and Margaret E Lundy. Uniqueness proof for a family of models sharing features of Tucker's three-mode factor analysis and PARAFAC/CANDECOMP. *Psychometrika*, 61:133-154, 1996.

[11] Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. *SIAM review*, 51(3):455-500, 2009.

[12] Yipeng Liu. Tensors for Data Processing: Theory, Methods, and Applications. Academic Press, 2021.

[13] Kristina Naskovska, Sher Ali Cheema, Martin Haardt, Bulat Valeev, and Yury Evdokimov. Iterative gfdm receiver based on the paratuck2 tensor decomposition. In *WSA 2017; 21th International ITG Workshop on Smart Antennas*, pages 1-7. VDE, 2017.

[14] Yannis Panagakis, Jean Kossaiti, Grigorios G Chrysos, James Oldfield, Mihalis A Nicolaou, Anima Anandkumar, and Stefanos Zafeiriou. Tensor methods in computer vision and deep learning. *Proceedings of the IEEE*, 109(5):863-890, 2021.

[15] Konstantin Usevich, Yassine Zniyed, Mariya Ishteva, Philippe Dreesen, and André LF de Almeida. Tensor-based two-layer decoupling of multivariate polynomial maps. In *31th European Signal Processing Conference (EUSIPCO 2023)*, pages 655-659, Helsinki, Finland, 2023.

[16] Shuochao Yao, Yiran Zhao, Aston Zhang, Shaohan Hu, Huajie Shao, Chao Zhang, Lu Su, and Tarek Abdelzaher. Deep learning for the internet of things. *Computer*, 51(5):32-41, 2018.

[17] Ke Zhang, Hanbo Ying, Hong-Ning Dai, Lin Li, Yuanyuan Peng, Keyi Guo, and Hongfang Yu. Compacting deep neural networks for internet of things: Methods and applications. *IEEE Internet of Things Journal*, 8(15):11935-11959, 2021.

[18] Yassine Zniyed, Konstantin Usevich, Sebastian Miron, and David Brie. A tensor-based approach for training flexible neural networks. In *2021 55th Asilomar Conference on Signals, Systems, and Computers*, pages 1673-1677. IEEE, 2021.