# The Effect of Tensor Rank on CNN's Performance

Eleftheria Vorgiazidou, Konstantinos Delibasis, Ilias Maglogiannis

► To cite this version:

Eleftheria Vorgiazidou, Konstantinos Delibasis, Ilias Maglogiannis. The Effect of Tensor Rank on CNN's Performance. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.549-560, 10.1007/978-3-031-34111-3_46. hal-05449809

HAL Id: hal-05449809

https://inria.hal.science/hal-05449809v1

Submitted on 9 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# The Effect of Tensor Rank on CNN's Performance

Eleftheria Vorgiazidou<fnref n="1" /> ID, Konstantinos Delibasis<fnref n="1" />(✉) ID, and Ilias Maglogiannis<fnref n="2" /> ID

**Abstract.** The goal of this work is to combine existing convolutional layers (CLs) to design a computationally efficient Convolutional Neural Network (CNN) for image classification tasks. The current limitations of CNNs in terms of memory requirements and computational cost have driven the demand for a simplification of their architecture. This work investigates the use of two consecutive CLs with 1-D filters to replace one layer of full rank 2-D set of filters. First we provide the mathematical formalism, derive the properties of the equivalent tensor and calculate the rank of tensor's slices in closed form. We apply this architecture with several parameterizations to the well known AlexNet without transfer learning and experiment with three different image classification tasks, which are compared against the original architecture. Results showed that for most parameterizations, the achieved reduction in dimensionality, which yields lower computational complexity and cost, maintains equivalent, or even marginally better classification accuracy.

**Keywords:** low-rank tensor · tensor decomposition · CNN

## 1 Introduction and Related Work

CNNs are one of the most important feature learning tools widely used in Computer Vision, Machine Learning and Artificial Intelligence in general. However, CNNs are considered computationally “heavy” systems as they require many arithmetic operations during convolutions and consume a large amount of memory. Concerns about the size and complexity of the models brought a consequent

This research has been co-financed by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH - CREATE - INNOVATE (project code: DFVA Deep Football Video Analytics T2EKΔΚ-04581).

<footnote n="1">¹ Department of Computer Science and Biomedical Informatics, University of Thessaly, Papasiopoulou Street 2-4, 35131 Lamia, Greece vorgiazidouele@gmail.com, kdelibasis@gmail.com</footnote>

<footnote n="2">² Department of Digital Systems, University of Piraeus, Karaoli and Dimitriou Street 80, 18534 Pireaus, Greece imaglo@unipi.gr</footnote>
demand for their simplification. One approach to reducing memory requirements and computational complexity is by compressing and decomposing convolution kernels.

After a thorough literature review, we noticed that few similar CNN architectures have been implemented, aimed to lower the rank of convolutional filters. Mamalet et al. [12] proposed the use of merged layers consisting of horizontal and vertical filters with a stride of 2 and separable filters, which aim to be low-rank from scratch. Rigamonti et al. [14] demonstrated that multiple filters can be replaced by fewer separable ones with rank equal to 1 resulting in negligible loss of accuracy. The same year, Denil et al. [2] represented the learnable weights' vector as a product of two low-rank components with smaller dimensions to reduce parameterization. Therefore, Denton et al. [3] used linear algebra and linear compression techniques, such as Singular Value Decomposition (SVD) [15], to yield low-rank convolution kernels while clustering the filters. Jaderberg et al. [7] proposed two different approximations both optimized based on reconstruction and empirical reconstruction filters' error maximum elimination. In the 1st the 2-D convolution filters are replaced by outer products of 1-D vertical and horizontal filters in the same CL, which is followed by a pointwise CL. In the 2nd approach, each CL is replaced by a sequence of two CLs, using vertical and horizontal kernels respectively, with a different number of filters.

Lebedev et al. [9] used the Canonical Polyadic Decomposition (CPD) [5] method to extract a sum of 4 equivalent rank-1 2-D tensors, which substitute the original convolution kernels consecutively. Tai et al. [16] tried to achieve the decomposition of 4-D tensors into a sequence of two horizontal and vertical low-rank convolution kernels and trained the deep networks with the reduced new parameters using batch normalization. Astrid et al. [1] also implemented the decomposition of kernels derived from fully connected layers using both CPD and SVD approach and developed an algorithm that intelligently chooses the tensor's rank. Phan et al. [13] rejected CPD and proposed an algorithm that uses either a corrected version of CPD or Tensor Kronecker Decomposition (TKD) method. Later, Lee et al. [10] proposed a non-common approach that builds a neural network consisting of 3-D rank-1 filters alternating with 1-D ones, which they were derived by decomposing the former, during the testing phase to speed up the process flow. These methods have been successfully applied to networks such as MobileNet [6], ResNet [4], yielding better results.

For this work, we researched and implemented a neural network architecture that uses, instead of one set of full rank filters, two cascaded 1-D filters. More specifically we investigated the mathematical properties and ability to classify images of a CNN with less memory requirements and computational cost, consisting of separable rank-1 filters. Our initial results of this application show that the equivalent CNN does not reduce the classification accuracy, although it is improved in terms of computational and memory requirements.
# 2 Methodology

## 2.1 Investigation of Convolutional Layer Decomposition

In this section, we adopt the method of Jaderberg et al. [7] and we decompose the CL with full-rank tensors into 2 CLs: the 1st one contains vertical convolutional kernels while the 2nd one horizontal convolutional kernels. We provide the mathematical formalism and we express the equivalent tensor as a sum of vertical and horizontal convolutions, investigating its mathematical properties. To this end we chose the well known AlexNet architecture [8]. Each filter of the AlexNet's 1st CL of 11 × 11 dimensions, is obviously a rank-11 matrix. We will show that it can be replaced by a sum of outer products of two separable ones, 11 × 1 and 1 × 11 that are rank-1 matrices.

Let an input (I) consisting of three channels $I_1, I_2, I_3$, ($C = 3$), be forward-passed in the 1st layer of the CNN. In the proposed implementation, the input layer of the neural network is followed by two CLs instead of one, to lower the rank of the equivalent kernels required for the convolution process. This method is applied to all CLs of the neural network. Figure 1 depicts the overview of the proposed workflow.

More specifically, let $F, C$ denote the filter size and the number of image channels respectively. The input is convolved with a number of $n_v$ vertical filters (or kernels), $K_1, K_2, ..., K_{n_v}$ of dimension $F \times 1 \times C$, each with $C = 3$ number of channels. Channel $c$ of kernel $i$ is denoted by $K_{ic}$. The channel $i$ of the output $A$ of the 1st CL is the following:

$$A_i = I_1 * K_{i1} + I_2 * K_{i2} + I_3 * K_{i3} + bk_i = \sum_{c=1}^{C} I_c K_{ic} + bk_i, \quad (1)$$

where $i \in 1, 2, ..., n_v$ and $bk_i$ stands for $K_i$'s bias. This output is convolved with the 2nd CL with $n_h$ horizontal filters of dimension $1 \times F \times n_v$ (the number of channels now is equal to $n_v$). Therefore, each kernel $O$ has a depth of $n_v$, with $O_{j,c}$ being the channel $c$ of the $j$th horizontal kernel, with $j = 1, ..., n_h$ and $c = 1, ..., n_v$. Thus, the output $B$ of the 2nd CL is the following:

$$B_j = A_1 * O_{j1} + A_2 * O_{j2} + ... + A_{n_v} * O_{jn_v} + bo_j = \sum_{c=1}^{n_v} A_c O_{jc} + bo_j, \quad (2)$$

where $bo_j$ stands for $O_j$'s bias.

Let us consider the equivalent low-rank tensor $T$ that is convolved with $I$ and generates the same output $B$, depicted as a workflow in Fig. 2. The dimensions of $T$ are $F \times F \times C \times n_h$, regardless the value of $n_v$. Further, let us name the slices of $T$ as $T_{jc}$, with $j = 1, 2, ..., n_h$ and $c = 1, 2, ..., C$.

Substituting Eq. 1 into 2 respectively results in a different representation of output $B$, which leads us to postulate a proposition about rank's computation for each $T_{jc}$ slice of the equivalent tensor $T$.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Low-rank layers architecture.</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Fig. 2. Procedure of B's generation through an equivalent tensor.</figcaption></figure>

**Theorem 1.** The rank of each slice $T_{jc}$ of tensor T is given by :

$$ \mathrm{rank}(T_{jc}) = \begin{cases} n_v, & \text{if } n_v < F \\ F, & \text{if } n_v \ge F \end{cases} \qquad (3) $$

*Proof. Generalizing Eq. 2 for any index j = 1, 2, ..., $n_h$ and substituting the expression for the output $A_i$, i = 1, ..., $n_v$ of the vertical layer from Eq. 1 yields the following expression for the output B of the horizontal layer:*

$$ B_j = (I_1 * K_{11} + I_2 * K_{12} + I_3 * K_{13} + bk_1) * O_{j1} \\ + (I_1 * K_{21} + I_2 * K_{22} + I_3 * K_{23} + bk_2) * O_{j2} + \dots \\ + (I_1 * K_{n_{v1}} + I_2 * K_{n_{v2}} + I_3 * K_{n_{v3}} + bk_{n_v}) * O_{jn_v} + b_{oj}. \quad (4) $$

Considering that the biases $b_{oj}$ of the 1st CL are equal to 0 and exploiting the distributive property, the Eq. 4, takes the following form:

$$ \begin{aligned} B_j &= I_1 * (K_{11} * O_{j1} + K_{21} * O_{j2} + \dots + K_{n_v1} * O_{jn_v}) \\ &\quad + I_2 * (K_{12} * O_{j1} + K_{22} * O_{j2} + \dots + K_{n_v2} * O_{jn_v}) \\ &\quad + I_3 * (K_{13} * O_{j1} + K_{23} * O_{j2} + \dots + K_{n_v3} * O_{jn_v}) + bo_j \\ &= I_1 \sum_{i=1}^{n_v} K_{i1} * O_{ji} + I_2 \sum_{i=1}^{n_v} K_{i2} * O_{ji} + I_3 \sum_{i=1}^{n_v} K_{i3} * O_{ji} + bo_j \\ &= \sum_{c=1}^{C} \sum_{i=1}^{n_v} K_{ic} * O_{ji} + bo_j. \end{aligned} \quad (5) $$

On the other hand, utilizing the definition of the equivalent tensor T, the output $B_j$ becomes:

$$ B_j = I_1 * T_{j1} + I_2 * T_{j2} + I_3 * T_{j3} + b_j = \sum_{c=1}^{C} I_c * T_{jc} + b_j, \quad (6) $$
where $b_j$ is the same bias as $b_{0j}$. Combining Eq. 5 and 6 we easily obtain

$$ 
\begin{aligned}
T_{j1} &= (K_{11} * O_{j1} + K_{21} * O_{j2} + \dots + K_{n_v 1} * O_{jn_v}), \\
T_{j2} &= (K_{12} * O_{j1} + K_{22} * O_{j2} + \dots + K_{n_v 2} * O_{jn_v}), \\
T_{j3} &= (K_{13} * O_{j1} + K_{23} * O_{j2} + \dots + K_{n_v 3} * O_{jn_v}),
\end{aligned}
\quad (7) $$

where $j \in 1, 2, \dots, n_h$. Finally, each slice of $T$ can be written as:

$$ T_{jc} = \sum_{q=1}^{n_v} K_{qc} * O_{jq}. \quad (8) $$

Thus, each tensor slice $T_{jc}$ is sum of rank-1 kernels, as each one's component is the convolution of two separable 1-D filters. Assuming that $n_v < F$ the denoted terms are $F \times F$ kernels with rank equal to $n_v$, as a sum of $n_v$ rank-1 components, otherwise, if $n_v \ge F$, the kernels are full-rank, with a rank of $F$, as the rank cannot overcome the kernel's dimension. So, each 2-D slice of $T_{jc}$ is a rank-$n_v$ matrix when $n_v < F$, or a rank-$F$ matrix when $n_v > F$. $\square$

## 2.2 Network's Architecture and Experimentation

The AlexNet CNN [8] consists of 5 CLs. All five model's untrained CLs are replaced by two successive layers, operating with rank-1 vertical and horizontal filters instead of full-rank ones. The 1st CL needs some additional modification, as it has filters with the largest dimensions, in contrast to the rest of the layers which contain filters of very small sizes. We experimented with the number of vertical kernels $n_v = 1, 2, 3, 4, 5, 8, 11, 20, 40$ and 96 used in this 1st layer, as it defines the rank of the equivalent tensor's slices, as proven in 2.1. The other CLs operated with $n_v = n_h$ kernels.

The proposed model incorporates half the corresponding original stride and padding values in each convolutional vertical and horizontal step, i.e. in the 1st CL of the original architecture, the stride value was 4, whereas each of the 2 CL with the vertical and horizontal kernels where set to stride =2. In case of padding equal to 1, the padding is divided to 1 and 0 in the two successive low-rank layers. The proposed architecture is depicted in Fig. 3, including all applied modifications and details about the stride, padding and size of each layer's output. It should be mentioned that no pretrained weights have been transferred to our implementation of AlexNet, in order to allow the experiments to emphasize on the differences between the architectures.
<figure><img src="image_5.png" /><figcaption>Fig. 3. Proposed AlexNet architecture.</figcaption></figure>

# 3 Results

## 3.1 Dataset Description and Preprocessing

The training and evaluation of the proposed model was performed using samples from 12, 20 and 30 different image categories, obtained from the freely available Caltech101 database [11], split into training, validation and test sets as shown on Table 1. An elementary transformation of the data into RGB images of dimension 227 × 227 is followed, simultaneously implementing data augmentation to the training and validation data, using randomly applied rotation, reflection and shear at both axes.

<table><caption>Table 1. Dataset's composition.</caption><thead><tr><td>Classes</td><td>Training Samples</td><td>Validation Samples</td><td>Test Samples</td></tr></thead><tbody><tr><td>12</td><td>2842</td><td>710</td><td>394</td></tr><tr><td>20</td><td>3207</td><td>801</td><td>445</td></tr><tr><td>30</td><td>3639</td><td>908</td><td>505</td></tr></tbody></table>

## 3.2 Network's Train Parameters

After experimentation, the mini-batch size was set to 8 images for every gradient update and the learning rate was set to 0.001, constantly. Stochastic gradient descent with momentum was selected as the optimization method and the maximum number of training epochs was set to 30, while the validation patience parameter was set to 10.
## 3.3 Quantitative Classification Results

According to Sect. 2 and the workflow scheme we listed in Fig. 1, the low tensor slice rank proposed architecture illustrated in Fig. 3 can be initialized with a user-defined $n_v$ value, which defines the rank of the convolution kernels. The achieved accuracy during test phase for three image classification tasks are averaged using the 10-fold cross validation technique and the standard deviation is computed. The network's performance was evaluated using a number of vertical kernels equal to $n_v = 1, 2, 3, 4, 5, 8, 11, 20, 40, 96$ compared with the original full-rank architecture's results. Three different experiments with increasing difficulty were performed, with 12, 20 and 30 image classes for classification. These results are summarized on Table 2, in comparison with the corresponding results of the original AlexNet architecture.

<table><caption>**Table 2.** Mean test accuracy and standard deviation for both $n_v$ low-rank and original architectures.</caption><thead><tr><th rowspan="2">Low-rank Architecture</th><th colspan="3">Classes</th></tr><tr><th>12</th><th>20</th><th>30</th></tr></thead><tbody><tr><td>n<sub>v</sub></td><td></td><td></td><td></td></tr><tr><td>1</td><td>82.64 ± 2.71</td><td>75.84 ± 3.04</td><td>67.81 ± 3.83</td></tr><tr><td>2</td><td>84.67 ± 3.65</td><td>75.79 ± 3.80</td><td>65.84 ± 2.67</td></tr><tr><td>3</td><td>84.11 ± 2.24</td><td>76.26 ± 3.24</td><td>68.17 ± 4.47</td></tr><tr><td>4</td><td>83.86 ± 3.71</td><td>77.52 ± 2.18</td><td>66.08 ± 5.54</td></tr><tr><td>5</td><td>84.95 ± 1.61</td><td>77.70 ± 3.60</td><td>66.31 ± 4.28</td></tr><tr><td>8</td><td>83.93 ± 2.89</td><td>76.89 ± 3.40</td><td>68.21 ± 3.75</td></tr><tr><td>11</td><td>84.19 ± 3.44</td><td>77.34 ± 4.11</td><td>67.24 ± 4.02</td></tr><tr><td>20</td><td>84.36 ± 2.83</td><td>75.25 ± 3.10</td><td>69.32 ± 2.23</td></tr><tr><td>40</td><td>84.80 ± 2.37</td><td>77.09 ± 2.06</td><td>69.26 ± 3.64</td></tr><tr><td>96</td><td>85.00 ± 2.56</td><td>77.79 ± 2.08</td><td>68.17 ± 3.90</td></tr><tr><td>Original Architecture</td><td>83.86 ± 2.61</td><td>76.17 ± 4.34</td><td>67.03 ± 3.44</td></tr></tbody></table>

As it can be observed in the results presented on Table 2, the accuracy achieved by the low tensor slice rank architecture shows a slowly increasing trend, for the first increasing values of $n_v$. In all three experiments, the low rank architecture marginally surpasses the original one in terms of classification accuracy for approximately $n_v \ge 2$. Finally, increasing the number of image classes clearly deteriorates the achieved accuracy, of both the original and the low rank architectures. Although one may have expected that the deep learning architectures would be almost unaffected by modest number of classes, it has to be mentioned that the deep neural networks that are used in this work have been trained from scratch on a rather small number of images. We refrained from using trained networks, since this would hinter the difference in performance of the architectures under comparison.
## 3.4 Arithmetic Complexity Reduction

Implementing the low-rank modifications, two CLs now operate with 1-D filters in a serial manner, instead of one CL with full rank 2-D filter, which results in reducing the complexity as follows. According to the Fig. 1, the CL operating with vertical separable filters presents $O(n_vCFHW)$ complexity, where $H, W$ describe the height and the width for input $I$ respectively. Similarly, the complexity of the layer operating with horizontal separable filters may be expressed as $O(n_hn_vFH'W')$, where $H', W'$ define the height and the width for the layer's input $A$. The overall complexity is calculated as

$$O(n_v(C + n_h)F(HW + H'W')).$$

If convolution's stride is set as $s = 1$, then $H = H', W = W'$, otherwise, if $s > 1$, then $H > H', W > W'$. So, for $s = 1$, the complexity appears as $O(n_vCFHW + n_hn_vFHW)$. Executing convolution with full-rank filters would yield a complexity result equal to

$$O(n_hCF^2HW).$$

The significance of this complexity reduction is demonstrated on Table 3, where the number of required arithmetic operations for both low tensor slice rank modified and original's full-rank layers are provided. So using separable filters reduces the computational cost along every CL as less arithmetic operations are required to perform convolution. More specifically, the differences in complexity are captured through a detailed example for multiple $n_v$ values and AlexNet's specific values $C = 3, n_h = 96, F = 11, H = 227, W = 227, H' = 109, W' = 109$ features.

The results on Table 3 present the rank of each $T_{jc}$ slice of the equivalent tensor utilized, the total low-rank complexity across every $n_v$ model and the difference between the original and the low-rank complexity for the 1st CL of AlexNet, along with the complexity's % reduction rate. The resulting complexity is reduced by 2 orders of magnitude (equivalently 1.7 billion less operations for forward pass through AlexNet) for $n_v = 1$. The $n_v = 96$ is the worst case scenario studied, achieving percentage reduction of 23.83% (equivalently 427,990,464 less operations).

In the same manner, we calculated the complexity for the 4 remaining CLs of AlexNet on Table 4, that are also constructed with a cascade of 2 CLs 1-D filters, but are not depended on the internal value $n_v$, comparing each with the corresponding original architecture complexity, in terms of their numerical and % percentage difference.

Overall, it is revealed that implementing a low tensor slice rank model would require millions less operations in every single CL, approaching a reduction of 8 hundred million operations in total when $n_v$ is equal to $n_h=96$ and surprisingly a reduction of 2 billion operations in total when executing a $n_v=1$ low-rank architecture. It is therefore clear that with such a huge reduction in arithmetic complexity and memory resources used during the convolution, we have created
<table><caption>**Table 3.** Number of arithmetic operations for different $n_v$ low-rank models for the 1st CL of AlexNet and the corresponding difference from the original layer's complexity.</caption><thead><tr><th>Low-rank Architecture<br>$n_v$</th><th>Rank of $T_{jc}$ Slice</th><th>Complexity</th><th>Reduction</th><th>% Reduction</th></tr></thead><tbody><tr><td>1</td><td>1</td><td>14,246,793</td><td>1,781,435,799</td><td>99.20</td></tr><tr><td>2</td><td>2</td><td>28,493,586</td><td>1,767,189,006</td><td>98.41</td></tr><tr><td>3</td><td>3</td><td>42,740,379</td><td>1,752,942,213</td><td>97.61</td></tr><tr><td>4</td><td>4</td><td>56,987,172</td><td>1,738,695,420</td><td>96.82</td></tr><tr><td>5</td><td>5</td><td>71,233,965</td><td>1,724,448,627</td><td>96.03</td></tr><tr><td>8</td><td>8</td><td>113,974,344</td><td>1,681,708,248</td><td>93.65</td></tr><tr><td>11</td><td>11</td><td>156,714,723</td><td>1,638,967,869</td><td>91.27</td></tr><tr><td>20</td><td>11</td><td>284,935,860</td><td>1,510,746,732</td><td>84.13</td></tr><tr><td>40</td><td>11</td><td>569,871,720</td><td>1,225,810,872</td><td>68.26</td></tr><tr><td>96</td><td>11</td><td>1,367,692,128</td><td>427,990,464</td><td>23.83</td></tr><tr><td>Original Architecture</td><td>11</td><td>1,795,682,592</td><td>0</td><td>0</td></tr></tbody></table>

<table><caption>**Table 4.** Number of arithmetic operations required for the remaining AlexNet's CLs according to both original and low-rank architecture.</caption><thead><tr><th>CL</th><th>Original Complexity</th><th>Low-rank Complexity</th><th>Reduction</th><th>% Reduction</th></tr></thead><tbody><tr><td>2</td><td>447,897,600</td><td>294,379,520</td><td>153,518,080</td><td>34.27</td></tr><tr><td>3</td><td>149,520,384</td><td>124,600,320</td><td>24,920,064</td><td>16.66</td></tr><tr><td>4</td><td>224,280,576</td><td>149,520,384</td><td>74,760,192</td><td>33.33</td></tr><tr><td>5</td><td>149,520,384</td><td>83,066,880</td><td>66,453,504</td><td>44.44</td></tr></tbody></table>

a much “lighter” neural system, that achieves the same or slightly improved classification accuracy.

## 3.5 Memory Requirement Reduction

Constructing a CNN with smaller filter dimensions in each CL can greatly reduce the number of learnable parameters. The learnable parameters derived from the original structure in total reach a number of 2,332,704, while the low-rank structure is defined by a much smaller $n_v$-dependent number of parameters equal to $1,650,688+1,089n_v$. Table 5 presents these results for different $n_v$ values, along with the absolute % difference from the original values. As it can be observed in Table 5 the difference in the total number of parameters for the low-rank architectures compared with the full-rank ones is quite significant, approaching 30%, which translates to approximately 680,927 for AlexNet. Despite the parameter reduction, the accuracy achieved by the low-rank architecture in every experiment is consistently not worse, or even marginally better than the original architecture.
<table><caption>**Table 5.** Number of learnable weights for both original AlexNet and low-rank architectures considering all CLs.</caption><thead><tr><th>Low-rank Architecture<br>n<sub>v</sub></th><th>Parameters</th><th>Reduction</th><th>% Reduction</th></tr></thead><tbody><tr><td>1</td><td>1,651,777</td><td>680,927</td><td>29.19</td></tr><tr><td>2</td><td>1,652,866</td><td>679,838</td><td>29.14</td></tr><tr><td>3</td><td>1,653,955</td><td>678,749</td><td>29.09</td></tr><tr><td>4</td><td>1,655,044</td><td>677,660</td><td>29.05</td></tr><tr><td>5</td><td>1,656,133</td><td>676,571</td><td>29.00</td></tr><tr><td>8</td><td>1,659,400</td><td>673,304</td><td>28.86</td></tr><tr><td>11</td><td>1,662,667</td><td>670,037</td><td>28.72</td></tr><tr><td>20</td><td>1,672,468</td><td>660,236</td><td>28.30</td></tr><tr><td>40</td><td>1,694,248</td><td>638,456</td><td>27.36</td></tr><tr><td>96</td><td>1,755,232</td><td>577,472</td><td>24.75</td></tr><tr><td>Original Architecture</td><td>2,332,704</td><td>0</td><td>0</td></tr></tbody></table>

# 4 Conclusions

In this work we investigated the mathematical properties of replacing a CL with full rank kernels by a series of two layers, each with 1-D kernels. From the proposed model we conclude that a reduction in the arithmetic complexity, computational cost and memory requirement of the resulting architecture can be achieved, without deteriorating its performance. As the rank of the $T_{jc}$ slice increases from 1 to 96, the difference in the number of arithmetic operations required compared to the original architecture's complexity decreases significantly. These conclusions suggest that the low tensor slice rank model can significantly reduce the complexity of the overall CLs by at least 747,642,304 operations with a maximum reduction of 2,101,087,639 operations. About memory requirements, the low tensor slice rank architecture reduces the number of parameters in the CLs compared to the original AlexNet architecture over half a million, at a percentage of 25-30%.

In further work we will study the mathematical properties of more low-rank neural architectures. We will also expand the datasets and the repetitions of the experimentation to establish statistical significance. Finally, we will investigate the applicability of this approach to deep learning architectures for object detection in video, which is a task that would greatly benefit from reduced arithmetic operations. Such an appropriate dataset is already available to our team in the DFVA Deep Football Video Analytics, as described in the Acknowledgments.
**Acknowledgements.** This research has been co-financed by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH - CREATE - INNOVATE (project code: DFVA Deep Football Video Analytics T2EKΔΚ-04581).

References

1. Astrid, M., Lee, S.I.: Deep compression of convolutional neural networks with low-rank approximation. ETRI J. 40(4), 421–434 (2018)

2. Denil, M., Shakibi, B., Dinh, L., Ranzato, M., De Freitas, N.: Predicting parameters in deep learning. Adv. Neural Inf. Process. Syst. 26 (2013)

3. Denton, E.L., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting linear structure within convolutional networks for efficient evaluation. Adv. Neural Inf. Process. Syst. 27 (2014)

4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

5. Hitchcock, F.L.: The expression of a tensor or a polyadic as a sum of products. J. Math. Phys. 6(1–4), 164–189 (1927)

6. Howard, A.G., et al.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

7. Jaderberg, M., Vedaldi, A., Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866 (2014)

8. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

9. Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I., Lempitsky, V.: Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6553 (2014)

10. Lee, S., Kim, H., Jeong, B., Yoon, J.: A training method for low rank convolutional neural networks based on alternating tensor compose-decompose method. Appl. Sci. 11(2), 643 (2021)

11. Li, F., Andreetto, M., Ranzato, M., Perona, P.: Caltech101. Computational Vision Group, California Institute of Technology (2003)

12. Mamalet, F., Garcia, C.: Simplifying ConvNets for fast learning. In: Villa, A.E.P., Duch, W., Érdi, P., Masulli, F., Palm, G. (eds.) ICANN 2012. LNCS, vol. 7553, pp. 58–65. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33266-1_8

13. Phan, A.-H., et al.: Stable low-rank tensor decomposition for compression of convolutional neural network. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12374, pp. 522–539. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58526-6_31

14. Rigamonti, R., Sironi, A., Lepetit, V., Fua, P.: Learning separable filters. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2754–2761 (2013)

15. Stewart, G.W.: On the early history of the singular value decomposition. SIAM Rev. 35(4), 551–566 (1993)

16. Tai, C., et al.: Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:1511.06067 (2015)