Cardinality Sparsity: Applications in Matrix-Matrix Multiplications and Machine Learning
Authors: Ali Mohaddes
Johannes Lederer
Venue: Accepted by TMLR ()

Abstract: High-dimensional data has become ubiquitous across the sciences but presents computational and statistical challenges. A common approach to addressing these challenges is through sparsity. In this paper, we introduce a new concept of sparsity, called cardinality sparsity. Broadly speaking, we define a tensor as sparse if it contains only a small number of unique values. We demonstrate that cardinality sparsity can improve deep learning and tensor regression both statistically and computationally. Along the way, we generalize recent statistical theories in these fields. Most importantly, we show that cardinality sparsity has a strikingly powerful application beyond high-dimensional data analysis: it can significantly speed up matrix-matrix multiplications. For instance, we demonstrate that cardinality sparsity leads to algorithms for binary-matrix multiplication that outperform state-of-the-art algorithms by a substantial margin. Additionally, another crucial aspect of this sparsity is minimizing memory usage. By executing matrix multiplication in the compressed domain, we can significantly lower the amount of memory needed to store the input data.

Published in Transactions on Machine Learning Research (08/2025)
Cardinality Sparsity: Applications in Matrix-Matrix Multiplications and Machine Learning
Ali Mohaddes Deportment of Mathematics University of Hamburg
ad:mohaddes@vni--hmmbyyy
Johannes Lederer
Department of Mathematics University of Hamburg
Reviewed on OpenReview: http:////ppeeeceeeeeeee/e//////s
Abstract
High-dimensional data has become ubiquitous across the sciences but presents computa- tional and statistical challenges. A common approach to addressing these challenges is through sparsity. In this paper, we introduce a new concept of sparsity, called cardinality sparsity. Broadly speaking, we define tensor as sparse if it. contains only small num- ber of unique values. We demonstrate that cardinality sparsity can improve deep learning and lensor regression both statistically and computationally. Along the way, we general- ize recent statistical theories in these fields. Most importantly, show that cardinality sparsity has strikingly powerful application beyond high-dimensional data analysis: it can significantly speed up matrix- cardinality sparsity leads to algorithms for binary-matrix multiplication that outperform state-of-theears algorithms by a substantial margin. Additionally, another crucial aspect of this sparsity is minimizing memory usage. By exccuting matrix multiplication the com- pressed domain, we can significantly lower the amount of memory needed to store the input data.
multiplications. For instance, we demonstrate that
1 Introduction
Contemporary data are often large and complex. On one hand, this offers the chance for extremely accurate and detailed descriptions of processes. On the other hand, inflicts computational obstacles and the risk of overfitting (Cybenko)] 1989). These computational and statistical challenges are under much investigation in
deep learning: [Cybenko| 1989). (Joshil 2022) (Srivastava et al.) 2014), (Mohaddes & Lederer 2023 (Ying,
2019h. Zhang et al. 2016 and (Bengio el al. 2015) but also in more traditional regression settings: Bartlet
et al. 2020). (Lederer 2021 (Eldar & Kutyniok 2012)
Thrampoulidis et al. 2015 and Elgobhary et al.
2016
A common remedy for these challenges is sparsity. Sparsity is usually defined as many zero-valued parameters
or many groups of zero-valued parameters: (El Karoui 2008), (Candès et al. 20066. Hebiri & Lederer 2020
and BBaiin] 1999). Sparsity has demonsttrated its benefits from mathematical: (Barron & Klusowskij 2018
(Neyshabur et al. 2015), (Taheri et al. 2020), Sechmidt-Hieber (2020), (Taheri al. 2021), (Bauer & Kohler
2019),
[Golestanch et al. 2024), (Mohades et al. 2014) and (Beknnzaryan 2022a), algorithmic: (Zhu et al.
2019
Lemhadri et al.] 2021), and applied perspectives: (Wen et al.] 20166) (Dong et al. 2011), (Iu et al.
2015 1, Christensen et al. 2009). Ravishankar et. al. 2019), (Sun & L. 2017 and (Akbari et al. 2022).
However, standard notions of sparsity have certain limitations. The primary drawback is their ineffectiveness in reducing computational costs. For instance, it is well known that algorithms for Boolean and binary matrix multiplications not faster than those for general matrix multiplications, with the most efficient. algorithms

---

Published in Transactions on Machine Learning Research (08/2025)
having a complexity of approximately O(n2-37) (Alman & Williams Lee 2002 As result, they do not fully exploit the potential computational benefits of sparsity. Moreover, they ask for zero-valued parameters or data points, while we might target values other than zero as well. A related limitation is that regularizers and constraints (often li-type norms) that induce zero-valued parameters also imply an unwanted bias toward small parameter values (eederer 2021 Section 2.3 on Pages 45ff).
This paper introduces a new notion of sparsity, cardinality sporsity, which beneficial for circumventing both computational and statistical challenges. This novel sparsity concept can enable rapid matrix multiplications, which are the primary source of computational issues in deep learning. Cardinality sparsity generalizes the idea of "few non-zero-valued parameters or data points" to "few unique parameter values or data points." Thus, cardinality sparsity provides a more nuanced and comprehensive notion of sparsity. It is inspired by four different lines of research: fusion sparsity (Tibshirani et al., 2005; Land & Friedman 1997), which pushes adjacent parameter values together; dense-dictionary coding (Elgohary et al.] 2016), which optimizes matrix-vector multiplications; layer sparsity (Hobiri & Lederer 2020p, which merges parameters of different layers across neural networks; and function approximations by neural networks with few unique parameter values (Beknazaryan 2022b). Theorems [ and 2 show that cardinality sparsity has essentially the same statistical effects in deep learning and tensor regression as the usual sparsity-yet without necessarily pushing parameters toward zero. Section B explains that cardinality sparsity can greatly reduce computational costs in pipelines that depend on large tensor-tensor multiplications, including once more deep learning and tensor regression.
Let us briefly elaborate on the computational aspect. The most consumnnn parts many optimization processes often matrix-matrix multiplications. The standard multiplication methods for M P and P N matrices have complexity of order O(MPN) (Papadimitriou 2003). However, many types of data enjoy cardinality sparsity: for example, image data often consist of only 256 different values. We can then reduce the complexity; indeed, we show that the number of multiplications in the matrix multiplication process can be reduced to OMMN from the compressed computation involves inspecting every element the resulting matrix.) with m and n the maximal number of unique values in the columns of the first matrix and the maximum number of unique values in the rows of the second matrix, respectively, that is, typically that assumed that the structure of the matrices is known. While the fast multiplication algorithm, with complexity of approximately (nn2.37). (Le Gall 2014 Alman & Williams), or O(n²) /af//(log (logn+ n²-5)), Yu 2015) remains applicable to any matrix, we focus on matrices with cardinality sparsity and known structure. By 'matrix structure," we refer to both the compressed version and the encoding matrix that we will introduce in Section This assumption relevant in neural networks and machine learning algorithms,, as the input matrix, which the primary source of sparsity, remains consistent across all learning iterations. Therefore, we can determine the input matrix structure through preprocessing before initiating the training process. It is important to note that this preprocessing step only needs to be performed once, incurring minimal time overhead, especially when dealing with large number of iterations. Additionally, this approach is advantagcous for Boolcan and binary matrix multiplication when the matrix contains only two distinct values. Moreover, another advantage of our method is that it remains valid even when one of the matrices is binary, whereas Boolean and binary multiplication methods usually require both matrices to satisfy binary conditions. This is particularly beneficial for example for categorical data when dummy encoding is applied. The key insight is to avoid redundant. multiplications, allowing us to effectively condense the matrices. Further details are provided in Section 3 For illustration, consider WV with
(The term MN ariscs because reconstructing the complete output
M and N. It should be noted
2.1
1.1
1
2.3
1
1.1
b
W =
V
2.1
1.1
c
d
d
3
2.3
3
4
2

---

Published in Transactions on Machine Learning Research (08/2025)
A naive multiplication of the two matrices requires 6 x 2 x 4 48 multiplications of entries. Using the cardinality trick instead, we can consider the compressed matrices
2.1
1.1
b d .
Cw
1
2.3
Cv
3
4
where Cw consists of the unique elements of the columns of W and Cy the unique elements of the rows of V. We can then compute CwCv and finally "decompress" the resulting matrix to obtain WV. This reduces the number of elementary multiplications to 3 x 2 x 2 12. Furthermore, the compression also reduces the memory requirements because the matrices Cw and Cv are considerably smaller than their original counterparts W and V.
Another crucial aspect our multiplication method its efficiency for matrices with a cardinality degree of order 2, such as binary (Boolean) matrices. Since the matrix consists of only two distinct values, pre- processing in this case is significantly simpler than in the general scenario. Furthermore, modifying the primary matrix to function
encoding matrix further streamline the pre-processing. This approach substantially accelerate the multiplication process for binary matrices, we will demonstrate in our simulations.
In addition, another major challenge in neural networks is the high memory demand for storing and processing input data. As we will sce, by employing cardinality sparsity, we can perform multiplication in the compressod domain. We only need to save the compressed version of the input data, which can decrease the required storage memory (Elgohary
al. 2016). The main reason is that we avoid saving all real numbers, each of which requires 32 bits to store. Further details will be explored in the simulation experiments (Section 6.1.2 Table 1 Our three main contributions are:
We introduce a new concept of sparsity, cardinality sporsity, defining a tensor sparse if it contains only a small number of unique values.
We demonstrate that cardinality sparsity can outperform state-of-the-art algorithms for matrix- matrix multiplications, including binary matrix-matrix multiplications, considerably. We also show that cardinality sparsity can reduce memory usage substantially.
Introducing new family of regularizers compatible with our notion of sparsity, and establishing cor- responding statistical guarantees, we prove that cardinality sparsity can reduce statistical complexity as well.
These contributions provide general framework for decreasing computational costs, reducing memory stor- age requirements, and addressing statistical complexity in machine learning systems and beyond.
The remainder of this manuscript is organized as follows. Section 2 recaps differret types sparsity and de- fines cardinality sparsity formally. Section 3 highlights the computational advantages of cardinality sparsity. discussed in Section and Section 5 where explore cardinality sparsity in the contexts of deep learning and tensor regression, respectively. Simulation results provided in Section G In Appendix A we review matrix-matrix multiplication algorithms, generalize our cardinality sparsity notion and review some technical results. The proofs deferred to the Appendix B
Our statistical results
2 Cardinality Sparsity
Spansity has become a standard concept in statistics and machine learning, arguably most prominently in compressed sensing (Eldar & Kutyniok] 2012) and high-dimensional statistics (Lederer) 2021). This section introduces a concept of sparsity that, broadly speaking, limits the number of different parameter values. In the subsequent sections, we will illustrate the potential of this concept from the perspectives of statistics (Section and Appendix 5) and computations (Section 4.2.1
3

---

Published in Transactions on Machine Learning Research (08/2025)
Modo-1 fiber
Mode-2 fiber
Mode-3 fiber
Figure Different fibers for a third-order tensor
We consider data-generating process indexed by an Nth order tensor A RQIxQ2xx..XxN The most standard notion of sparsity concerns the number of nonzero elements: A. is k-sparse if
AAlo == #supp[A] S k,
where supp denotes the number of non-zero elements of the tensor. Instead, we count the number of different parameter values in each fiber. Fibers the sub-arrays of tensor whose indexes, all but one, are fixed. More precisely, the mode-n fibers are the set of all sub-arrays (vectors) that generated from fixing all but the nth index (Kolda & Bader 2009 Bowen & Wang 2008). Indeed, by Aq.... " -a-1 "...111....""" .g we mcan vector of length Qn where di is lixed for Vi E { NEW (n). The fibers of a third-order tensor are illustrated Figure 1 Now we are ready to provide the definition for our notion of sparsity:
Definition 1 (Cardinality Sparsity). We say that the tensor A E xQxx...Qx is (n, (n,k))spp ks-arss if
k 2 max { ( uo.
vQ.) Ag Ant11.9999
E {l,..., Qi...... (L)...,Q1)...... In-1 E fl...., Qa-1h.
d0+1 E {1, Qu+1). IN (I,...., (wt....QQtt.
In other words, tensor is (n, k)-sparse if the mode-n fibers have at most k different values. We call this notion of sparsity, cardinality sparsity.
We can also define cardinality sparsity for vectors, which actually are special cases of tensors. Cardinality sparsity for vectors related to the fusion sparsity notion (Tibshirani et al. 2005 Land & Friedman, 1997). They assumed that the vectors are pice-wise constant. However, the main difference that they assumed that the adjacent elements are similar. While in the cardinality sparsity, look for similaritics in the entire vector. Similar to tensors say that vector = (V1...., "T,) has cardinality sparsity if:
#(V1 v..) < Jn.
Another special case that interested in is sparsity in the matrices. We say matrix has column-wise cardinality (mode-1 cardinality) sparsity if it has repeated values in each column. For example, the most sparse case (in the sense of this paper) is matrix whose columns' elements are unique as follows:
b
b
W =
a
c
4

---

Published in Transactions on Machine Learning Research (08/2025)
Similarly, row-wise (mode-2) sparse matrix is matrix whose rows have cardinality sparsity property.
We now introduce an alternative definition of sparsity that is more convenient for analysis. This new defini- tion allows us to concentrate on standard sparsity rather than cardinality sparsity, In order to provide this definition we first review some preliminaries. Consider a tensor A R x xxxxxxxxXxxxx Xx.xx. The mode-n product of tensor A TRQ XRxxxQxx.xQ by a matrix U E XaQx is a tensor B RRQ 1eQxx..xQ== X Q xQx denoted as:
B A U,
where each entry of B is defined as the sum of products of corresponding entries in A and U
gu.... Da 1N
Σ 9o
Bugi..... Un
Also, we define a difference lensor A as follows:
= Ax, A,
(1)
where the matrix A is of size (2:) x On such that for each row of A, there is one element equals to 1, one clement equals to 11 and the remaining (Qn 2) clements are zero. This means, that instead of studying mode-n cardinality spansity of a lensor, we can focus on the standard sparsity of the difference tensor.
Similarly, we can say matrix W mx has cardinality sparsity if the matrix resulting from the difference of each two elements in each column is sparse in the standard sense. Therefore, difference matrix. The difference matrix could be considered as the product of two matrices A and W. Where again, the matrix A is of size (2) x m such that for each row of A, there is one element equal to 1, one element equal to -1, and the remaining (m - 2) elements are zero. For more clarification, we provide the below example.
concentrate the
Example 1. As an example, assume that m = n = 3 and wrile
w11
W12
W13
W
W/21
W/22
U/2
W31
W32
W33
where Wij IR for all i,j (1,2,3). According to the construction of the matrix A mentioned above, can set
-1
A
0
1
-1
-1
0
and hence difference matrir W is equal to:
W11 W21
W12
1/22
W/3
11233
W AW
W21
W31
W22
W32
W23
W33
W31
Wi
W32
W12
W33
W13
We can observe that this transformation helps us to measure the similarities between any two rows of each matrix. In particular, if all of the rows of W are the same, then we obtain that AW 0. Therefore, cardinality sparsity could be induced by employing matrix A.
3 Cardinality Sparsity in Computational Cost Reduction
As we mentioned another important aspect of cardinality sparsity is reducing the computational cost. For the sake of completeness, we provide a matrix-matrix multiplication method that reduces computations for matrices with cardinality sparsity. This method inspired by the matrix-vector multiplication approach studied in (Elgohary et al. (2066). The matrix-matrin multiplication would appear in many applications. In neural network, we need to perform chain of matrix multiplication. For the case when the input
5

---

Published in Transactions on Machine Learning Research (08/2025)
is matrix, can employ matrix-matrix multiplication in each step to find the output. Morcover, in tensor analysis, lensor-matrix products also could be computed by some matrix products. Therefore, an efficient matrix-matrin multiplication method can reduce computational costs in many applications. In the next part, we employ cardinality sparsity to provide a matrix multiplication method with low computational complexity.
3.1 Matrix Multiplication for Sparse Matrices
In this part, we review the necessary background to introduce our matrix-matris multiplication algorithm. The elements of a matrix W E MxP which is sparse in our sense, can be described as follows
Wiy E (a Amyki
where (1,..., P) is fixed and m & M is the maximum number of unique elements in each column (where for simplicity we did not change the order of unique elements in each column). We define an indicator function that denotes the positions of cach repeated clement in a fixed column as follows
I(ns) = t; E (1,...m.)
If the number of unique elements in column was 8 less than we then put m-- remaining elements are equal to zero. Therefore, we obtain a matrix I E gMXP whose elements are integers from the set {0, 1.... .m.. Consider matrix-vector multiplication W x v where v = [e1.-.. " To perform this multiplication it is enough to consider the unique elements of each column of W and compute their multiplication with the corresponding clement of Finally, we use the matrix I reconstruct the final result.
3.1.1 Matrix-Matrix Multiplication Using Cardinality Sparsity
To develop an efficient matrix-matio matrix multiplication method, we consider two scenarios. We first assume that P not very large compared with M and N. The key point for the matrix-matrix multiplication mothod (for small) to employ the below definition.
Definition 2. Consider the mudliplication between WMXP and VpxN. In this case, we have
P WxV = Σ
W & Vo
(2)
where (i) is the ith colurnn of the matrix W and V(A) is the ith row of V, and each ( 8 V(a is M rank-one matriz, computed as the tensor (outer) product of two vectors.
This definition can lead us to a matrix-matrix multiplication method with less number of multiplications. We again assume Wi, {ai...... amigh when j e (I.... ,P} is fixed and m M the maximum number of unique elements in each column of W. Also we assume that új E 1b., binj) where j E (1,...,,. is fixed and is the maximum number of unique elements in each row of V.
Our goal is to find method in which we can generate elements in Equation 2 with less number of computations. We then compute the summation over these elements. Similar to the previous method we define two matrices I and as follows
I(ar,)) = i; ( k; k
(1, "
=
Now we compute the tensor product between two vectors Ow = =aa1.j ... am,jl and CV 1by, b (vectors obtained from unique elements of the jth column of W and jth row of V). For reconstrueting the jth element Equation (2) we use the aforementioned tensor product and the Cartesian product (which in this paper we show by x.) between jth column of and jth row of J. This Cartesian product denotes the map which projects elements of Cw & Cv to the matrices of size M x N. In the below we provide a numerical example to illustrate the procedure.

---

Published in Transactions on Machine Learning Research (08/2025)
Example 2 (Fast multiplication for sparse matrices). Let
2.1
1.1
2.3
1
1.1
b
a
W
;V =
2.1
1.1
d d
c
2.3
4
We compute the first element in Equation as on enample. We have W = 2211 1 1 2.1
3 3
T
and Va) - -Ia a a]. In order to implement the proposed method we compule Cwa) = [2.1 and CV.,) aa b] Also by deftnition of and J we obtain firs column of I and the first row of J is equal
3 T
to ((1) = [1 2
3 3] and J(n)
1 2 1] respectively. The tensor product of Cwas and
Cva, is equal to
2.1a
2.1b
D
b
3a
35
Now we fond the Carlesian product of T()) and ((1) map Dj to (
V(L) as follows
(1,1)
(1.1)
(1,2)
(1,1)
(2,1)
(2,1)
(2
(2,1)
70 X Ja)=
(2,1)
(2.1)
(2,2)
(2,
(1,1)
(1,1)
(1,2)
(1,1)
(3,1)
(3,1)
(3,2)
(3,1)
(3,1)
(3,1)
(3,2)
(3,1)
This helps us to map D2 to ( & V(n- The second term in Equation can be obtained similarly.
The Cartesian product is just concatenation of elements and does not increase compulational complexity. This new matrix- matrix-matris multiplication procedure can lead to computations of order O(mPn) which could be much less than usual matrix multiplication methods (O(MPN); & M,n N).
This multiplication method is particularly useful when P is not a large number. However, P is large, a different approach more effective. In this case, we use the usual definition of matrix-matrix multiplication, i.e. we consider each element of the multiplication result as the inner product of rows of the first matrix and columns of the second matrix. This approach lets us reduce computational costs. In this case, generate matrix I and Cw for the first matrix so that we perform encoding row-wise. In order to obtain the (i, element of the multiplication result we consider the inner product of the ith row of the first matrix and the jth column of the second matrix. In order to perform factorization, we employ the ith row matrix I and perform summation over elements of the column of the second matrix based on the ith row I. For example, we have
Cwe) = [1.1 2.3 2.3
4, := 2122 2
Then we sum the first, second, and fourth elements of the ith column of the second matrix and multiply by the first element of Owiy. We also sum the third and fifth elements of the ith column of the second matrix and multiply by the second element Cwia and finally sum these two results. Algorithms 2 and g explain these multiplication methods, while the standard multiplication algorithun reviewed in Algorithm
Example 3 (Fast binary mairix multiplication). Now we provide crucial ezample of binary malrices. The key point that in the binary case, since there are only two possible values, the preprocessing process is much simpler than for the general case. To perform preprocessing, we need to find the compressed matria and the encoding matriz. For matrices with column- wise cardinality sparsity, we store the first row, and the second of the compressed matria is simply the complement. of the first row, resulting in the compressed

---

Published in Transactions on Machine Learning Research (08/2025)
matrix. The encoding matrix is obtained by storing columns of the original matrix that start with zero and the complements of those that start with one. This melhod streamlines the preprocessing process and leads to further computational reductions.
For example, in the following, you can observe a binory matriz along with its equivalent encoding and compressed matrix:
0
0
0
1
0
!
A
Ta
1
Ca
0
0
0
0
0
0]
[
0
1
The rest of the maltiplication process is simslar to the general case.
We omit the multiplication steps to
prevent redundoncy.
The experiment for this section are presented in Figures
2 3
5
G
and 14
4 Cardinality Sparsity in Deep Learning
In (Hebiri & Loderer 2020), the authors investigated the notion of layer sparsity for neural networks. Cardinality sparsity analogous layer sparsity, but instead addresses the sparsity in the width of the network. Moreover, recent work function approximation (Beknazaryan, 2022b) has demonstrated that deep neural networks can effectively approximate broad range of functions with smaller set of unique parameters. This provides further motivation to focus on cardinality sparsity as means of reducing the network's parameter space and increasing its learning speed.
this section, we will first review the neural network model, as well as some notations and definitions that will be necessary. We will then study a general family of regularizers that includes both the cardinality and & regularizers as special cases. While the standard & regularizer promotes sparsity but does not explicitly discourage large values, this broader class of sparsity-inducei regularizers can also help control large magnitudes in the solution. Finally, will provide prediction guarantee for the estimators of a network with cardinality sparsity, and study the computational aspects of the the introduced regularizer.
We focus on regression rather than classification simply because regression is the statistically more challenging problem: for example, the boundedness of classification allows for the use of techniques like McDiarmid's inequality (MeDiarmid] 1989 Devroye et al. 2013
4.1 Deep Learning Framework
In this part, we briefly discuss some assumptions and notations of deep neural networks. Consider the below regression model:
= g [ma]
+ ue
E (1).....)
for data (y/, 21 21).... (Yn, Fn) IR Rd Where g Idd + R is unknown aata-generating function and ; "..
R is the stochastic noise. Our goal is to fit the data-generating function with feed-forward neural network ge Rd + + R
ge [2] = Whrh [.......... ............ W
a E Rd..
indexed by :
= (WL, ..... WO). Where L a positive integer representing the number of hidden layers and W' E BP++1 Xp: are the weight matrices for (0....,,J L} with Po d and PL+1 1. For each E (O...., L), Re++ are the activation functions. We assume that the activation functions
the functions f IR satisfy P(O Op, and are aLip Lipschict continuous for constant aLip [0, oo) and with respect to the Euclidean norms on their input and output spaces:
[+[=] +|='|1 < anip l= 2'll
for all z, E RR++
8

---

Published in Transactions on Machine Learning Research (08/2025)
Furthermore, assume that the activation function fl RP++ - R is non-negative homogeneous of degree (compared to (Taheri et al. 2021b)
f[sz]
sf[z] for all s E (0,00) and IR++
A standard example of activation functions that satisfy these assumptions are ReLU functions 2021 We also denote by
(Theei et al.
A = { == := w w R Xp:
as the collection of all possible weight matrices for ge. Taking into account the high dimensionality of A Pt (n, o0)), and following (Taheri et al. 2021 we can employ the below estimator:
(R, (2) E arg min
(u xgol>:l) + dr
xe)0,00) DEA
n2
where E [0, 00) is tuming parameter, A + R indicates the penalty term and the corresponding unit ball denotes by Ag. = [O] < 1}. This regularization enables us to focus on the effective noise: == supnEA, 12/n Li=1 0a [u// which is related to the Gaussian and Rademacher complexities of the function class Gb = (oo E A₂)- We need to show that the effective noise controlled by the tuning parameter with high probability. In order to consider the control on the effective noise similar to (Taheri
et al. 2021 we define:
A E min {8 E (0) P(z, S 21
which is the smallest tuning parameter for controlling the effective noise level t. We consider the prediction error
err[xpa] ==
(glæ)) - xan [x..))
1=1
for k E [0, oo), S2 E An
Furthermore, assume that noise variables are independent, centered, and uniformly sub-Gaussian for
constants K,7 E (0, oo) (van de Geer 2000 page 126); (Vershynin 2018 Section 2.5):
K22
1x,1 R Ee
1) 5
max (E (E....)))))
In addition, we give some definitions which are convenient for illustrating the main results of this section. For vector and a weight matrix Wi say
Ix||
Σ i=1
|2 12/2//.
and
pi+1 P
[w'l,
Whyl
(3)
We also need the usual matrix norm Definition Suppose that -laa
Geijn 2014
page 5):
R and
1.|+ Rm + R are vector norms. For E Rmxn we define:
Bx
B
max
pERm((0)
|

---

Published in Transactions on Machine Learning Research (08/2025)
Let I- la is norm R", ||r is norm R" and | l. is norm on I" then for any a E IR:
|BCal [ |B+++/CCall s [BI+=
We now introduce some other parameters which are used in the main result of this section. We set
G (G y... G"),
(4)
and
MG == max | 11-11
(5)
where G); j E (0,... L are non-invertible matrices and GP is the invertible matrix induced by the matrix G¹, and ils inverse malrix denotes by G-1 Im(G') - k
4.2 A General Statistical Guarantee for Regularized Estimators
Regularization plays essential role in sparse regimes. Sparsity may lead to degenerate or unstable solutions, since multiple sparse parameterizations can yield the same mapping. In this section we propose regularizer
which resolves this by stabilizing the solution space, reducing redundancy, and enforcing structured sparsity.
To induce cardinality sparsity, as shown in Equation O A. Where we define A == (A4, ..... A"), for
one can promote standard sparsity AO for
e = (W Wo) for W' e RP44XM,
which belongs to parameter space:
A == ((WL,... ....., Wo): w RP--1XP().
Indeed in this case, we again consider neural network with hidden layers. For (0....,, L}, our first step to transform each weight matrix into difference matrix, W to A'w'. Where similar to Equation (1). we define A' E R x)pe+1 so that cach row of A' corresponds to distinct pair of indicos (i,j) with s < < pt41- In the row associated with (i,,), the entry in column is set to 1, the entry in column j is set to 11, and all remaining PI+1 - entries are zero. Thus, A' enumerates all possible index pairs, yielding (P rows in total.
The term |AO|, penalizes absolute differences, encouraging many of them to be zero and thereby promoting equality among paired elements. Consequently, this term enforces cardinality sparsity in the matrices O meaning that each column of ④ contains repeated elements. The sparsest case occurs when each column contains only single unique value.
However, a limitation arises because ||0|] alone is not a norm: vanishes for any constant-per-columm matrix (i.e., AO = for all such in the kernel of A). This allows arbitrarily large constants to pass without penalty, potentially leading to degenerate solutions. Moreover, since the measure is based on differences of rows, distinct matrices O can yield the same value of AQ.
To address these issues, we introduce second term, +|I where II is the projection onto ker(A). This term ensures that the projection of θ onto the kernel, namely, the component corresponding to matrix with a single repeated element per column, is also sparse. In addition, the inclusion of this term guarantees that the overall rogularizer is a norm (Proposition []
This family of regularizers can be viewed as a generalization of the standard & regularizer. While the usual l penalty does not prevent large values, this form of sparsity control also mitigates the effect of large magnitudes. Accordingly, the regularizer can be expressed as follows:
|40| VIII
(6)
where A A + ker(A) is the projection to ker( (A) and positive constant.
10

---

Published in Transactions on Machine Learning Research (08/2025)
In this paper, we focus general statistical guarantee for regularized estimators, of which cardinality- based sparsity and the standard notion of sparsity are only two special cases. We consider regularizers of the form
b/ = |G0| + v|1l6|11 for o
(7)
where G E RNP: (N 2: P. == 1 pi) is an (invertible or non-invertible) matrix, v E (0, 00) is a constant, and 1l. E [dii((er(G))xXP. the projection onto ker(G), the kernel of G. One special case is l 2-regularization, where G = Another special case of the regularizer is the one that induces cardinality sparsity, where =
Studying the effective noise introduced in the previous section enables us to prove the main theorem of this section. We show that the Ab.t must satisfy Ap. < 26. Where will be explicitly defined in Equation LT.. Combining this with Theorem 2 of (Taheri et al. 2021 we can prove the below theorem for prediction guarantec of the introduced regularizer.
Theorem (Prediction guarantee for deep learning).
Assume that
V2
> C(Mc + 24
L
(8)
Vlg((P)) log(2m) Vni
where C E (0, oc) is a constant that depends only on the sub-Gaussian parameters K and y of the noise. Then, for n large enough, and v 2 1/(1 Ma)
err erT"(Kn
inf {err"kgga, 2Ax].
xE(0,00) E.A.A
with probability at least 1/n.
The bound in Theorem establishes essentially (1/Vn)-decrease of the error in the sample size n, logarithmic increase in the number of parameters P, and almost exponential decrease in the number of hidden layers L if everything else is fixed. In addition, we know that aLip is equal to one for ReLU activation functions. This theorem is valid for any matrix in general and we just need to adjust parameter Mc. We continue this part with two corollaries that result from Theorem 1
Corollary 1 (Standard fi-sparsity). Let G= O then Mo = 0, and for == 1 the regularizer in 7 would be
changed & reggularizer. In this case, the Equation 11 for large number of parameters could be written as follows
V2 T,
21-1
log 2n Vn.
A z ((aTap)2
Vlog 2P
(9)
and then, for large enough,
2 err
Ki KKnBn
<
inf 0.),)0))
ferr? [Kpa,] 1 2Ax
Qg EAAp
with probability at least 1/n.
This corollary states that & l-regularrzerr is special case of regularizer in Equation 77 This corresponds to the setup in Taaheii et al. 2021). Another exciting example of matrix G is the matrix A. Before illustrating the next corollary for cardinality sparsity, we provide the below lemma.
Lemma 1 (Upper bound for MA). Let G = A. Then we have
2
MA < sup PP, (E(
(10)
where P3 denotes the numbbe of parometers of each weight matria.
11

---

Published in Transactions on Machine Learning Research (08/2025)
This lemma implies that, the value of MA is always less than two. Moreover, in practice, deep learning problems contains many parameters and as result VPj is large number thus MA is real number close to zero (MA < 1).
Corollary 2 (Cardinality sparsity). Let G = A then MA & 1, and in this case, the Equation 11 for large number of porameters could be written follows
V2 2L-1
log 2n
A 2 220
log 2P
(11)
vn
and then, for n large enough, and v 2 1/(1-MA)
inf {err"kga,, +22x). kE(U,0)) A,E.An
with probability at least 1/n.
4.2.1 Further Insights into Cardinality Sparsity Regularization
Now, we further investigate another aspect of the introduced regularizer. Recall that b[O] | + v 11I[ where, θ
Wo) for W E R In the above equation the first term is l regularizer, the second term however, (TA+) needs more investigation. We know that the kernel space of A' is a matrix as follows (elements in each column are the same)
B1
B2
Bn
B1
B2
8n
B1
B2
3n
Therefore, the projection of matrix W on this space, i.e. 11 would be of the below form
2011 - W0g wa
M011 20g 1122
22 1009
where wys are entries of matrix W As result, calculating II is not computationally expensive and we just need to compute mean of each column. Statistical benefits experiments are presented in Figure 11
Also note that from Equation 6 we apply the regularizer below in the cardinality-indueing case,
[[ = |40|1 + 11111
Since according to Proposition the above equation is norm, it is consequently convex. Therefore, our regularizer remains convex, in contrast to direct cardinality penalties, which are indeed non-convex and combinatorial. Our approach leverages norm regularization to promote sparsity in a tractable manner, avoiding the computational difficulties of solving combinatorial problems.
In addition, to further mitigate computational complexity, one approach is to apply the projection Ps onto the space of matrices with cardinality degree k. Within the learning algorithm, sparsity by mapping the updated weight matrix of the hidden layer onto cardinality-spane matrix via a structured projection process. Specifically, for each column of the matrix, we first sort the values in ascending or deeccending order. Then, we partition these sorted values into k distinct groups, where k denoles the predetermined sparsity degree, controlling the level of sparsity in the resulting matrix. Subsequently, we compute the mean of the values within each partition and replace all original values that partition with this mean, effectively making all values within each partition uniform. This projection ensures that the matrix retains sparse structure while preserving essential information.
enforce cardinality
12

---

(x, 1 = XY. Published in Transactions on Machine Learning Research (08/2025)
5 Sparsity in Tensor Regression
In this section, we first provide a brief review of tensors and tensor regression. Then, we study cardinality sparsity for tensor regression. For two tensors E R4x xxxxxxxxg Pi...P II...., IK. Pi...., PL are positive integers) and Y E xxxxxx..PPxxxxx................. product
(P1,..., PL. Qu are positive inlegers) the contracled lensor (x, 1) E Rh xxxxxxxxxxxxxxx. x..xxxx..x...
can be defined as follows
((X, V) 1
Pi
Ai AK Pr. pL Y PL...P..
P1=1
P==1
In the special case for matrices X E RRXP and Y E ERP we have
Assume that we have n = 1, I,...,N .N observations, and consider predicting tensor Y E IxQx X. from a tensor X E NxPx x Pa with the model
Y = X B) + U,
where B XxxxxxxXxxxxxxxxxxXX is coefficient array and U P.NXQIx--XXM is array. The least-squares solution is as follows
B E arg (X, L)
where || denotes the Frobenius norm However, this solution is still prone to over-fitting. In order to address this problem an L2 penalty on the cocfficient tensor B, could be applied
B E arg miggmi//V.... IY (X, L) A|BIP
(12)
B
where À controls the degree of penalization. This objective is equivalent to ridge regression when W E Nx1 is a vector and E NP matrix. Similar to the neural networks we modify Equation (12) so that it is proper for cardinality sparsity follows
arg (X, + Ab[B],
(13)
and similar to the previous case again define
b[
IB x, G|11 1I1
(14)
where again G is any matrix, 1l. the projection to ker(G) and constant. Then we have the below theorem for tensors.
Theorem 2 (Prediction bound for tensors and boundedness of effective noise). Consider the Equation
130. Let U. U.p...p. o(),o2). Assume effective noise 2 2||(U,X] x) bl is bounded, then for every B e
XPxxX.x...x.. it holds that
B), (X, B)
inf B) (X, A) + 226[A]}
Also let Uppier sn (0,000)) Then the effective noise 2%, 1/o is bounded with high probability, i.e.,
p > U) 2 1-P .... ....) Que - (2ott)))
where is defined as the follows
sup
2 21 x) qu 4M -w.-iirrrr-e 9M I/N
91 E(l...... -.) Qx....QQx)
13

---

Published in Transactions on Machine Learning Research (08/2025)
Note that, for proving this theorem we just employed the norm property of b[B), which is irrelated to form of matrix G. Therefore, can immediately result the below corollaries.
Corollary 3 (Prediction bound for tensors and boundedness of effective noise for L (irreuuaarizer). Let 0
then regularizer in 14 changes to E1-regularizer and by assumptions of Theorem 2 it holds that
1xx,B) X L) S inf [nf (X. A) 2Xb[A]}
and the effective noise x U) 1o0 is bounded with high probability.
Corollary (Prediction bound and boundedness of effective noise for cardinality sparsity). Let G - A then regularizer in 14 induces cardinality sparsity and by assumptions of Theorem 2 it holds that
B), - (x, B) S inf - A), - -
&
and the effective noise (x,, 10 bounded with high probability.
Corollary 5. In the special case, for Y == X where X E RNNM, U RRix, the Theorem would change to
Pf 2 21, 2 1 - Pie -Re 2o0ix=)
and s changes to
= sup (1((X, pt.p1 I/N.
mE(l,.... P1)
This result matches Lemma 4.2.1 of (Lederer 2021
6 Expremintal Support
In this section, we initially present experimental evidence demonstrating the advantages of sparsity in reduc- ing multiplication and memory costs. Subsequently, we implement cardinality sparsity in machine learning systems. Our results underscore the substantial benefits of cardinality sparsity, demonstrating its advanta- geous impact.
6.1
Exprimental Support (Part
This section validates our results. First, we Lest the methods for malrix-matrix multiplication introduced in We evaluate our approach for both simulated and real-world datasets.
Section 3.1
6.1.1
Matrix-Matrix Multiplication
This part evaluates the performance of the proposed multiplication methods. We consider the matrix mul- tiplication task AB; A E RMXP, B E RPN, We generate random matrices A, B with varying degrees of sparsity, that is, varying number of unique elements of columns of A and rows of B. Specifically, we popu- late the coordinates of the matrices with integers sampled uniformly from {l....,k) k). where degree, and then subtract a random standard-normall generated value (the same for all coordinates) from all coordinates (we did this subtraction obtain a general form for the matrix and avoid working with matrix with just integer values). We set the sparsity degree 10 and generated square matrices of different sizes. We then compared method (Algorithm 3 with both the Strassen algorithm (Strassen 1969) and the naive approach [Algorithn[]) p for matrix multiplication. The empirical results demonstrate large gains in speed especially for matrices of large size. As ratustrated in Figure 2 our muluipicattinn technique surpasses both Strassen's 's and the conventional algorithms when dealing handling large matrices. For smaller matrices, the preprocessing cost relatively high. However, for larger matrices, the significant reduction in redundant multiplications makes our algorithm more efficient. Consequently, our algorithm outperforms those that do not optimize for redundant multiplications.
the sparsity
14

---

Published in Transactions on Machine Learning Research (08/2025)
Fast, Standard and Strassen Multiplication
Fast, Standard and Strassen Multiplication
Fast, multiplicaiinn
2'
Fast: multiplicaion
Standard multiplication
Standard multipliction multiplication
Strocsen multiplictiion
27
400
27
time
log
200
21-
100
2-1
100
200
300
100
500
600
700
800
900
100
200
300
400
500
600
700
800
900
Square matrix size
Square matrix size
(a) Square nonbinary matrices multiplication without scaling.
(b) Square nonbinary matrices multiplication with loga-
rithmic scaling.
Figure 2: Algorithm compared to the naive Algorithm 1 and Strassen Algorithm for nonbinary matrices of sparsity degree equal to 10. Our algorithms are faster than the naive and Strassen approach, especially for large sizes.
6.1.2 Binary Matrix Multiplication
In this section, will concentrate on the binary matrix multiplication of square matrices, implementing preprocessing similar to example 3 We compare our approach with the method described by Strassen (Strassen 1969) and the standard multiplication technique. Similar to the previons part we randomly generated square matrices sizes suitable for Strassen's 's algorithm. We utilized the algorithu referenced in to perform our matrix multiplication. As shown Figure our multiplication method significantly performm both Strassen's and the standard algorithms.
Fast, Standard Strassen Multiplication
Fast, Standard Strassen Multiplication
Fast multiplication
Fast multiplication
500
Standard multiplicaiion
Standard multiplication
Stressen multiplication
Stressen multiplication
scale
2'
tine 19)
300
2²
200
2a.
100
2-2
2-4
1000
200
400
600
700
800
900
100
200
400
600
700
800
900
Square matrix size
Square matrix siz
(a) Square binary matrices multiplication without scaling.
(b) Square binary matrices multiplication with logarith- mic scaling.
Figure 3: Algorithm 3 compared to the naive Algorithm and Strassen Algorithm for binary matrices. Our algorithms are considerably faster than the naive and Strassen approach.
6.2 Matrix Multiplcation for Real-world Datasets
In this section we will annalize cardinality sparsity for real-world datascts. We conducted additional ex- periments on real-world datasets further validate the efficiency of our approach. Specifically, we applied multiplication method to the real datasets (Leter Recognition Dataset, Letter Digits Dataset, and Firm
15

---

Published in Transactions on Machine Learning Research (08/2025)
Teacher Clave Direction Classification Slate 1991 Alpaydin 1998
Vurka 2011
and compared the results
with the standard and Strassen multiplication method.
Matrix multiplication for real datasets commonly occurs in applications such as neural networks, where input data must be multiplied by weight matrices. To evaluate our method, we selected a dataset as input and multiplied with randomly generated weight matrices. For example, the size of the input dataset in the Letter Digits is 1797 64. Using our matrix multiplication algorithm, we performed operations to multiply the imput dataset (1797 64) with weight matrices of size 64 We varied the value of 'n' to test method under different conditions, simulating various numbers of columns (or number of nodes in neural networks). Subsequently, we mcasured the runtime, including preprocessing time.
Our results reveal a significant gap between the runtime of our method and that of standard matrix mul- tiplication algorithms. We also applied our method to the "Firm Teacher Clave Direction Classification" dataset (Vurka 2011), where dummy encoding usod. We demonstrated that datasots utilizing dummy en- coding could achieve substantial performance improvements with our matrix multiplication algorithm. Since matrix multiplications the primary computational workload in many applications like neural networks, optimizing them significantly reduce energy consumption when processing real data.
Letter Recagnition Dataset (20000 16)
Letter Recognition Dataset (20000 16)
Fast multipliration
120
Standard multiplicatio
Strassen multiaiicttion
Time (s) flo scale)
(log
30/
multiplicat on Standard multplication Strassen nultiplication
40
80
130
Number nodes
Number nodes
(a) Matrix sizes: 20000 x 16 and 16 x n.
(b) Matrix sizes: 20000 x 16 and 16 x n
Figure 4: Panel (a) displays the normal scale, while Panel (b) shows the log scale. Our algorithms significantly outperform both the naive and Strassen methods terms of speed for the Letter Recognition Dataset.
Moreover, we carried out more simulation experiments to evaluate our "binary multiplication algorithm" against some modern statc-of-the-art methods. Note that many modern state-of-the-art algorithms are pri- marily theoretical, with no publicly available implementations. We selected two approaches that are more practical to mmplement-nanmne ison. As shown in F our algorithu outperforms the Four Russians algorithu and achieves comparable performance the bit- packed approach, which leverages CPU-level parallelism and low-level optimizations. This further demonstrates that our algorithm can significantly reduce CPU usage and hardware resource consumption.
the Four Russians algorithm and the bit-packed method for direct compar-
We also conducted additional ablation experiments by varying the sparsity degree while keeping the matrix size fixed at 512 x 80 and 80 x 512. Specifically, we increased the cardinality (i.e., the number of unique ele- ments per row or column) to observe its impact on computation speod. As expected, performance docreasod with higher degrees of sparsity. Figure illustrates the results.
6.2.1 Memory reduction using cardinality sparsity
In this part, we explore memory optimization through cardinality sparsity in real-world datasets. Since we perform multiplication in the compressed domain, only the compressed version of the input data needs
16

---

Published in Transactions on Machine Learning Research (08/2025)
Letter Digits dataset (1797 x 64)
Tast multippication
letter Digits datacet (1747 x 64)
Standard multiplication
multiplicaiion
Strassen multplication
Standard multipicction
Strassen multp nultipication
6X12
Time (log scale)
(s) (log scale)
4X12
150
Number
(a) Matrix sizes: 1797 x 64 and 64 n.
(b) Matrix sizes: 1797 64 and 64 x n.
Figure 5: Panel (a) displays the normal scale, while Panel (b) shows the log scale. Our algorit significantly outperform both the naive and Strassen methods terms of speed for the Letter Digits Dataset.
clave-direction class dataset (10787 70)
clave-direction class dataset (10787 x0)
Fast multipiization
Standard mutiplicationn
Strassen multipliction
Time (s) (log scale!
Time (5) (log scale)
Fast multipicatoo
Standard multplication
nultiplication
40
10
70
80
100
80
70
80
30
100
Number of nodes
Number of nodes
(a) Matrix sizes: 10787 x 20 and 20 n.
(b) Matrix sizes: 10787 x and 20 n.
Figure 6: Panel (a) displays the normal scale, while Panel (b) shows the log scale. Our algorithms significantly outperform both the naive and Strassen methods in terms of speed for the Clave-direction Class Dataset.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Performance vs. Matrix Siee
Cardinality
12
BE Packed
10
100
200
300
430
500
600
700
800
900
Matrix Size
Figure 7: Runtime comparison of our binary multiplication algorithm with the Four Russians and bit- sit-packed methods square binary matrices of varying sizes.
Performance vs. Cardinality Degree
Cardinality
Standard
Straceon
3-
Cardinality Degree
Figure 8: Computation time vs. cardinality for fixed 512 x 80 and 80 x 512 matrices. As the number of unique elements increases, traditional methods begin to outperform the cardinality sparsity approach.
longer need to store all the real numbers, which typically require 32 bits each. By eliminating the duplication of real numbers, we only need integers in the encoding matrix to reference the locations of these unique values. we provide examples using real datasets: the Yeast dataset (Nakai 1991), the Concrele Compressive Strength dataset (Yeh) 1998). and the A141 2020 Predictive Maintenance Dataset (ai4 2020), to demonstrate the efficiency of cardinality sparsity in reducing memory usage in neural networks. These datascts are referred to as data 1, data 2, and data 3, respectively, in the table.
to be stored, significantly reducing memory requirements. This is because
6.3 Additional Empirical Results
We continue this section by presenting additional empirical results that further substantiate our methods and thcorics. particular, we provide further experiments investigate the performance of the matrix-matrix multiplication techniques introduced in Section 3.1 We then apply cardinality sparsity to neural networks and tensor regression. Finally, illustrate the regularizer statistical benefits.
18

---

Published in Transactions on Machine Learning Research (08/2025)
We compared the memory requirements in bits with and without the use of cardinality sparsity in bytes. This table illustrates that implementing cardinality sparsity significantly reduces memory costs in a neural network.
Table 1:
Input data
Using sparsity
Without sparsity
data 1
9752.5
47488
data 2
19662.8
37080
data 3
125496.0
480000
Again consider the matrix multiplication task AB; AER AMRM B BERPX, We generate random matrices A, B with varying degrees sparsity, that is, varying number unique elements of columns of A and rows of B. Specifically, we populate the coordinates of the matrices with integers sampled uniformly from (1....,k) k)), where k the sparsity degree, and then subtract random (aandard-normall generated value (the same for all coordinates) from all coordinates (we did this subtraction to obtain a general form for the matrix and avoid working with a matrix with just integer values). We then compare our methods (Algorithms 2 and 3 with the naive approach (Algorithm 1 for the corresponding matrix multiplication. The factor by which our methods improve on the naive approach's 'es speed averaged over 10 times generating the matrices are presented in Figure g There are two cases: (i) P < M, where we use our Algorithm 3 (Figure Panel a); and (ii) P > M, N, where we use our Algorithm large gains in speed especially for small sparsity degrees.
2 (Figure Panel b). The empirical results demonstrato
30
30
25
25
Speed Raiio
Speed Ratio
20
20
10
10
Same Same.speed
Same speed
100
125
150
175
200
126
150
175
200
Sparsity Degree
Spanitt Degree
(a) Matrix sizes: 2000 x 40 and 40 x 2000.
(b) Matrix sizes: 100 x 10000 and 10000 x 100.
Figure 9: Algorithm 3 (Panel (a)) and Algorithm 2 (Panel (b)) compared to the naive Algorithm Our
algorithms are considcrably faster than the naive approach, especially for small degrees of sparsity.
6.3.1 Cardinality Sparsity in Neural Networks and Tensor Regression
We study cardinality sparsity in machine learning systems using both real and simulated datasets. For the neural-networks application, we use the Optical- Reciical-Rcoonnttion-off ofrc Honduritten Diggit data (Dua & Graff] 2017). We train two-layer relu network with 40 nodes in the hidden layer with gradient descent. For backpropagation,
After each weight use Algorithm update, we project the weight matrix of the hidden layer onto cardinality-spare matrix: We sort and split each column into h partitions, where l is the sparsity degree. We then replace the old values by the mean of the values in each partition (see our Section (.2.1), so that all values in each partilion are equal after the projection. The accuracy of the neural networks is illustrated in Figure I0] This result shows that, cardinality sparsity training accelerates the parameter training substantially. is important to understand that it possible conduct the experiment without using weight matrix projection. This is because the input data, which has high dimension, is sparse, and most of the computational efficiency comes from
use Algorithm [ for forward operations;
19

---

Published in Transactions on Machine Learning Research (08/2025)
the sparsity of the input data. Although weight projection may further decrease computations, it's not mandatory.
We also investigated the accuracy versus time in two cases. As we observe in Figure 10 our method obtains high accuracy rates faster than the usual training methods.
100
With sparsity
90
Without sparsity
Training Accuracy
60
40
2000
4000
6000
8000
10000
12000
14000
Time
Figure 10:
Training accuracy of the network trained with (red) and without (blue) cardinality sparsity as a lunction of training time. The cardinality sparsity accelerates the parameter training.
We also consider the tensor regression. We applied the Tensorly package for this simulation Kossaaff et al. 2016). In order to employ our matrix-matrix multiplication method, we reshape the tensor into a matrix. We assumed that the input matrix is random and sparse in our sense (data are generated similar to section 6.1.Ip. The input is a tensor of size 1000 x 16 16 and we applied iterations for learning. Regression with sparse multiplication and standard multiplication takes 18.2 and 35 seconds respectively. Therefore, employing sparsity can lead to faster regression process. Note that, in the tensor regression process only substitute standard multiplication with sparse multiplication therefore, the accuracy rate is the same in both experiments.
Now we examine the performance of the regularizer and use experiments to certify the introduced regularizer can increase network performance. We consider lcast-squares minimization complemented by the regularizer which induces cardinality sparsity. We trained a layers neural network where the number of the first, second, third, and fourth layer are equal 2, 10, 15, and 20 respectively, where we employed Inverse Square Root Unit (x/vI+ + )² identity, arctan, and again identity activation functions. The samples are generated by standard normal distribution labeled by weight parameters which are sparse in sense, plus standard Gaussian noise. We set the target weights' sparsity equal to 4. We trained the network with 1000 iterations. As shown in Figure [ incorporating regularizer driven by cardinality sparsity can enhance regression performance compared to Lasso regression. We also computed the Frobenius norm of the difference between trained weights and the target weight. The difference networks with and without regularization are equal 64.24 and 76.12 respectively which is less value in a network with regularization we expected. Note that all simulations are performed by Macbook Pro laptop with 8 cores of CPU and 16 Gigabytes of RAM.
To further assess the generalizability and practical benefits of our approach, we conducted experiments using the letter recoguition dataset as input to a neural network. In this setting, we evaluated performance using the mean squared error as the primary metric. Our objective was to determine whether the advantages of cardinality pparsity-previousj] demonstrated matrix multiplication and simulation bisks-would also extend real-world machine learning workflows. The experimental results in Figure 12 indicate that incor- porating cardinality sparsity not only preserves model accuracy but also leads to notable improvements in
20

---

Published in Transactions on Machine Learning Research (08/2025)
Lasso regularizer
Cardinality regularizer
function
10
-10
-20
-4
-2
0
2
3
Figure 11: This figure compares deep learning with cardinality and standard sparsity. Integrating a regu- larizer based cardinality sparsity improve regression performance in comparison to Lasso regression. Cardinality sparsitiy (orange line) makes the vanilla estimator (red line) smooth and brings it closer to the true function (black line) especially in the first part of the domain.
computational speed. This suggests that the method is effective in accelcrating neural network training and inference across different types of datasets.
Meen Squared Error Time (Aligned Start)
Derpriment Mutleal
Weln
00108
Figure 12: We also evaluated our method on the letter recognition dataset by feeding it into neural network and measuring the mean squared error. The results show that cardinality sparsity can enhance processing speed across different datascts while maintaining performance.
6.4 Cardinality Sparsity Beyond Machine Learning
This section provides additional simulations to further illustrate the effectiveness of our approach, considering the setting where the matrix structure is assumed to be given. We emphasize that preprocessing time is always included in our simulations, and we never rely on the assumption that the matrix structure is known beforehand.
One of the most important cases where this assumption is natural when we need compute powers matrir. In this setting, preprocessing only needs to be done once, after which the encoded structure
21

---

Published in Transactions on Machine Learning Research (08/2025)
be reused for every multiplication. This property appears in many applications, including dynamical systems (Hirsch et al. 1974) and graph theory (West. 2001), making it particularly strong use case for our cardinality sparsity multiplication algorithm, especially when the matrix is sparse in the cardinality sense.
For instance, in graph theory, the k-th power of an adjacency matrix reveals the number paths of length & (West] (001). To test this, we generated a random adjacency matrix with 200 nodes and compared our cardinality sparsity multiplication algorithm against the standard dense multiplication. As shown in algorithm achieved much faster runtime.
Figure 13
Another important application where the structure of the matrix can be assumed fixed is in the construction of multivariate Markov chain models. Here, the transition matrix remains constant across iterations. We applied our algorithm in this conlext as well, and as illustrated in Figure 14 the cardinality sparsity multipli- cation method again performed very well, highlighting its usefulness beyond machine learning settings (Ching
& Ng. 2006).
50
Cardinality sparsity
Standard multiplication
Time (seconds)
20
1.0
1.5
2.0
2.5
3.0
3.5
5.0
Adjacency Matrix k-th Power (A*)
Figure 13: Comparison of cumulative runtime between the proposed cardinality sparsity multiplication algorithm and the standard dense matrix multiplication for successive powers of random adjacency matrix with 200 nodes. The adjacency matrix is sparse in the cardinality sense, and the experiment demonstrates that the proposed method achicves significant computational savings.
Acknowledgments
This rescarch was partially funded by grant 520388526 (TRR391) by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation).
References
A14I 2020 Predictive Maintenance Dataset. https://doi.org/u 2ttpg/11..44450 24.24432.22242./ 44422/C5HSCCC C5HS5C.
UCI Machine Learning Repository, 2020. DOI:
P. Akbari, M. Gabriel, and C. MacKenzie. Retrieving and disseminating information about disasters through natural language processing tools. In 1IE Annu. Conf. Proc., pp. 1 6. Inst. Ind. Syst. Engrs. (USE)), 2022.
J. Alman and V. V. Williams. A refined laser method and faster matrix multiplication. Proc. 2021 ACM- SIAM Symp. Discrete Algoritims (SODA).
E. Alpaydin. Optical recognition of handwritten digits. UCI Machine Learning Repository (ac- cessed via seikit-learm, 1998. URL https://archive. ics uei ucc.eddd///mm/aaaasettttt..aa
22

---

Published in Transactions on Machine Learning Research (08/2025)
100
Cardinality sparsity
Standard multiplication
Time (seconds)
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
State step k
Figure 14: Application of the cardinality sparsity multiplication algorithm to a multivariate Markov chain model with a fixed transition matrix. The proposed method again outperforms the standard dense multipli- cation terms of runtime.
S. Bakin. Adoptive regression and model selection in data mining problems. Ph.D. thesis, The Australian National University, Canberra, Australia, 1999.
A. Barron and J. Klusowski. Approximation and estimation for high-dimensional deep learning networks. arXiv:1809.0330 03090, 2018.
P. Bartlett, P. Long, G. Lugosi, and A. Tsigler. Benign overfitting in linear regression. Proc. Natl. Acad. Sei. USA, (17(48):30063-30000 30070, 2020.
B. Bauer and M. Kohler. On deep learning as a remedy for the curse of dimensionality in nonparametric regression. Ann. Stat., 2744):2261-228) 2019.
A. Beknazaryan. Neural networks with superexpressive activations and integer weights. Intell. Comput. Comput. Conf., pp. 445 451, 2022a.
A. Beknazaryan. Function approximation by deep neural networks with parameters +1,2) J. Stot. Theory Pract., 2022b.
E. Bengio, P. Bacon, J. Pineau, and D. Precup. Conditional computation in neural networks for faster models. arXiv:151.00629 2015.
R. Bowen and C. Wang. Introduction to Vectors and Tensors. Courier Corporation, Chelmsford, Mas- sachusetts, 2008.
E. Candès, J. Romberg, and T. Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory, 52(():489-55) 509, 2006.
W. Ching and M. Ng. Markov Chains: Models, Algorithms and Applications. Springer, New York, 2006.
M. Christensen, J. Ostergaard, and Jensen. On compressed sensing and its application to speech and audio signals. Conf. Record 43rd Asilomar Conf. Signals, Systems, Comput., pp. 356-360, 2009.
G. Cybenko. Approximation by superpositions of sigmoidal function. Math. Control Signals Systems, 2 (4):303 314, 1989.
L. Devroye, L. Györfi, and G. Lugosi. Probabilistic Theory of Pattern Recognition, volume 31. Springer Sci. Bus. Media., New York, USA, 2013.
23

---

Published in Transactions on Machine Learning Research (08/2025)
W. Dong, X. Li, L. Zhang, and G. Shi. Sparsity-based image denoising via dictionary learning and structural clustering. CVPR, pp. 457-464, 2011.
D. Dua and C. Graff. UCT machine learning repository. University of California, Iroine, School of Information and Computer Sciences, 2017. URL http::/ ics. uci. edu/mll
N. El Karoui. Operator norm consistent estimation of large-dimensional sparse covariance matrices. IEEE Trans. Inf. Theory, 36(6):2717-2766, 2008.
Y. Eldar and G. Kutyniok. Compressed sensing: theory and applications. Cambridge University Press, 2012.
A. Elgohary, M. Bochm, P. Haas, F. Reiss, and B. Reinwald. Compressed linear algebra for large-scale machine learning. Proc. VLDB Endow., 9(12):960-99) 971, 2016.
M. Fu, B. Zhao, C. Carignan, R. Shosted, J. Perry, D. P. Kuehn, 7. Liang, and B. Sutton. High-resolution dynamic speech imaging with joint low-rank and sparsity constraints. Magn. Reson. Med., 73(5):1820 1832, 2015.
R. Geijn. Notes on vector and matrix norms. Dept. of Computer Sci., Univ. of Texos at Austin, 2014.
P. Golestanch, M. Taheri, and J. Lederer. How many samples are needed train deep neural network? arXiv:2405.10 16696, 2024.
M. Hebiri and J. Lederer. Layer sparsity in neural networks. ar arXiv:2006. 156044 2020.
M. W. Hirsch, R. T.. Devaney, and S. Smale. Differentiol Equations, Dynamical Systems, and Lineor Algebra, volume 60. Academic Press, New York, USA, 1974.
A. Joshi. Perceptron and neural networks. Machine Learn. Artif. Intell,, pp. 57 72, 2022.
T. Kolda and B. Bader. Tensor decompositions and applications. SIAM Rev., 51(3):455 500, 2009.
J. Kossaifi, Y. Panagakis, A. Anandkumar, and M. Pantic. Tensorly: Tensor learning in python. arXiv:1610. 09555, 2016.
S. Land and J. Friedman. Variable fusion: new adaptive signal regression method. Tech. Rep. 656, Dept. Stat., Carnegie Mellon Univ., 1997.
F. Le Gall. Powers of tensors and fast matrix multiplication. Proc. 39th Int. Symp. Symbolic Algebraic Comput., pp. 296-303, 2014.
J. Lederer. Fundamentals of High- Siahh-Dimnsional Statistics. Springer, New York, 2021.
J. Lederer. Statistical guarantecs for sparse deep learning. arXim:2912.05427, 2022.
L. Lee. Fast context-free grammar parsing requires fast boolean matrix multiplication. J. ACM, 49(1):1 15, 2002.
Lemhadri, F. Ruan, and R. Tibshirani. Lassonet: Neural networks with feature sparsity. Tnl. Conf. Artif. Intell. Stot., pp. 10 18, 2021.
C. McDiarmid. On the method of bounded differences. Surv. Comb., 141(1):148 188, 1989.
A. Mohaddes and J. Lederer. Affine invariance in contimuous-lomain convolutional neural networks. 1.02042.11.09255. 2023.
Mohamad M. Mohades, Ali Mohades, and Aliakbar Tadaion. A Reed-Solomor Code Based Measurement Matrix with Small Coherence. TEEE Signal Processing Letlers, (7))):839 843, Jul 2014.
Kenta Nakai. Yeast. UCI Machine Learning Repository, 1991. DOI: http:///doo.or/1002444/// 24..2443/C55K6..
24

---

Published in Transactions on Machine Learning Research (08/2025)
B. Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control in neural networks. In Proceedings of the 28th Conference on Learning Theory, pp. 1376-1401, Jun 2015.
C. Papadimitriou. Computational complexity. Encycl. Compul. Sci. pp- 260 265, 2003.
S. Ravishankar, Ye, and Fessler. Image reconstruction: From sparsity to data-adaptive methods and machine learning. Proc. IEEE, 108(():86-100)) 109, 2019.
J. Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. Annals of Statistics, 48(4):1875 -8997 2020.
D. Slate. Letter recognition. UCI Machine Learning Repository, 1991. URL https://archiver ics. aci. wwweed///m..
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-1955) 2014.
V. Strassen. Gaussian elimination is not optimal. Numer. Math., 13(4):354 356, 1969.
W. Sun and L. Li. Store: sparse tensor response regression and neuroimaging analysis. J. Mach. Learn. Res., 18(1):4908 4944, 2017.
M. Taheri, N. Lim, and J. Lederer. Efficient feature selection with large and high-dimensional data. arXiv:1609.077 07195, 2020.
M. Taheri, F. Xie, and J. Lederer. Statistical guarantees for regularized neural networks. Newral Netw., 142: 148 101-161,110 2021.
C. Thrampoulidis, S. Oymak, and B. Hassibi. Regularized linear regression: A precise analysis of the estimation error. In Proceedings of the 28th Conference on Learning Theory, pp. 1683 1709, Jun 2015,
R. Tibshirani, M. Saunders, S. Rosset, J. Zhu, and K. Knight. Sparsity and smoothness via the fused lasso. J. R. Stat. Soc. Ser. B, 67(1):91 108, 2005.
Vaart and Wellner. Weak convergence. In Weak Converyence and Empirical Processes, pp. 16-28. Springer, New York, USA, 1996.
S. van de Geer. Empirical Processes in M-Estimation, volume 6. Cambridge Univ. Press, Cambridge, UK, 2000.
R. Vershynin. Dighe-Dimmsionall Probability: An Introduction with Applications in Data Science, volume 47. Cambridge Univ. Press, Cambridge, UK, 2018.
M. Vurka. Firm-teacher clave-direction classification. URL https://archive. ics uci http:///eeeee.aaaaaaea/aataattea
UCI Machine Learning Repository, 2011.
C classification
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. Adv. Neural Inf. Process. Syst., 29:2074 2082, 2016.
D. West. Introduction Graph Theory. Prentice Hall, New Jersey, 2001.
C. Xu and Z. Zhang. Random tensors and their normal distributions. orXiv:1908.0111 2019.
C. Yeh. Concrete Compressive Strength. UCI Machine Learning Repository, 1998. wtt.e//rr/.....////////
DOI:
X. Ying. An overview of overfitting and its solutions. J. Phys. Conf. Ser., 1168:022022, 2019.
Huacheng Yu. An improved combinatorial algorithm for boolean matrix multiplication. In International Col- loquium on Automata, Languages, and Programming (ICALP), volume 9134 of Lecture Notes in Computer Science, pp. 1094-1105, 2015.
25

---

Published in Transactions on Machine Learning Research (08/2025)
Y. Zhang, J. Lee, and M. Jordan. Ll-regularized neural networks are improperly learnable in polynomial time. In Proceedings of the 33rd International Conference on Machine Learning (ICML.) pp. 993--101, Jun 2016.
M. Zhu, T. Zhang, 7. Gu, and Y. Xie. Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks modern gpus. Proc. IEEE/ACM Int. Symp. Microarch., pp. 359-37 371, 2019.
26

---

Published in Transactions on Machine Learning Research (08/2025)
A
Appendix
In this section, we first explore matrix-matrix multiplication algorithms. We then broaden our concept of sparsity. Subsequently, we provide additional technical results. The implementation of our main algorithm (Algorithm provided in the final section.
A.1 Algorithms
In this section we review matrix-mant matrix multiplication algorithms.
Algorithm Standard Multiplication
Require: E M.MxPBB E Ensure: AB
function STANDARDDMULLTPPICAATOOAAA B) for to M do
for j 1 to N do
Output[(,,]] + for k + 1 to P do Output [. += A(i k] * B[k,
end for end for end for return Output end function
Algorithm Multiplication by Cardinality Sparsity (P> M,
Require: W E MxP. V E RPN
Ensure: WV
1: function SPARSEMULTPPLIAATOOOYWWWW V) for j 1 P do
Cwa) <- unique elements of column WU) ([wi wm.sl
[w: i; E (1.... m
end for
6: for k t- 1 to M do
7:
for j - to N do
8:
for - to do
Output[k: [k: - sum (sun [[[[(, =
== i,j) * Cw [k, i))
end for 10: end for
11:
12: end for
13: return Outpul
14: end function
27

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm 3 Multiplication by Cardinality Sparsity (P < M,
Require: W& E MxPP; ERPN
Ensurc: WV
function SPARSEMULZTPPICATOOOOW,, V) for 1 to P do
Cwo) < unique elements of column WU) ([wi, ... wm.!')
I [wx, < i; iE (1,...,)) m
end for
for j f 1 to P do
Cvn f unique elements of row Vos (10g v)
[| + i; ie [1, 11, n}
end for
for f 1 to P do
Dj t- Cway @ CV.
for i to m do
for + to n do
Auxiliary[[ jl == i, J[j, 1= k]+ k)
end for end for Output < Output + Auxiliary end for return Output end function
A.2 Extending Our Notion of Sparsity
Now we recall some notations and definitions. sparsity.
We then provide a generalized setting for our notion of
A.2.1 Some Notations and Definitions
We recall that
M = Mc == max 1G' I1-
(15)
where G the invertible matrix induced by G¹, We also define below auxiliary parameter for our compu- tations
b = max 1M.
(16)
Furthermore, for the sake of simplicity from now on, we depict I and W by θ and W respectively. Indeed
:= 0+ (WL.
WD)
(17)
where Ô ker(G) and θ belong to the space which is orthogonal to the ker(G). Subsequently, for each E L), we decompose each weight matrix W according to the kernel of G denoted by ker(G'). Specifically, we can write W == W W where all columns of W belong to ker(G") and all columns of W are within the space, which orthogonal ker(G') denoted by ker(G)) This implies that W = G'Wi = G'W + G'W = G.W. We can also rewrite 7 like the below:
6,,,6
|G0|, + v/
28

---

Published in Transactions on Machine Learning Research (08/2025)
Employing Equation
3
equivalent to
7
write the penalty term
RRT P ΣΣΣ (G
PLL MEΣ
[
W
(18)
Note that, we could also study the performance of regularizer in neural networks away with the scaling parameter (see (Lederer 2022) In the next part, we provide generalization of our sparsily concept.
A.2.2 Sparsity Notion Extension
In more general setting, we can fix a partition for the weight matrix so that elements with the same value are in the same group. Indcod, we assume that in the sparsest While in the previous case there is one individual row in the entire matrix in the sparsest case. Indeed, the generalized setting concerns block sparsity, while in the normal case, the entire matrix is considered as one single block. The group sparsity concept can be employed to study this sctting. In this section, we apply the regularizer based on the penalty term defined in equation to the analog of group sparsity.
each partition contains unique row.
First of all, for E L), we let I == {B)..... BBd be a partition of {L...., where E 21. is the number of partitions within I. We further write the network weights row-wise
W ((W:) ... -(W (
and define
W! B = ((W)) ... (Wore
) i di)
fiala
where 1p:j] = B and BB] is the cardinality of B for all t0...., L}. As before, there exist matrices A E (x(11))x11)) such that A/W gy is matrix whose rows row-wise subtractions of elements of W Bq. Let A be the matrix constructed by these sub-matrices in such way that
d Σ
VP. 2
Aw Σ
A W 88
Pr.
|
Σ &.EB
w.
Wth,,
where A Mpst18p5 Misxpy and Mmxa denotes the space of matrices. Moreover, Pajj, is the number of parameters in group where lj is calculated from summing over all 2-combinations of the cardinality of partitions, i.e.
B (1B8) 1)
2
In order to clarify more consider the below example.
Example Let B (1,2,3), By 54.5) and let Wi 555 then PL, 15 and P2, 10 and A W'W is follows
29

---

Published in Transactions on Machine Learning Research (08/2025)
V15
V
15
0
0
15
15
V15
0
15
0
0
V10
V /10
W11
W12
W13
W14
W15
221
W222
W23
222
W25
131
W33
W33
W34
W35
W41
12
W43
W/41
W/5
W51
W52
U55
W54
W55
V15 w
W2)
V15 w2 wy)
VI5 w1 w)y
/10 ws)
where W1,'." Wy denote the rous of the second matriz. In addition, we have
15
V15
V15
/5
Al
[v10 -v10]
V55
0
V15
Note that the computations for this sctting are similar the previous casc, where we considered non- invertible matrix G for our analysis in the last Section. The only change related to the computation of the upper bound for the parameter MA, which is addressed in the next corollary.
Corollary For the matria A with generalized cardinality sporsity the value of MA bounded as the below
MA = SA 11-11 S sup IAA 11-1
sup sup
VP.
A.3 Technical Results
In this part we provide some technical results which required to prove the results of this paper.
Convexity
We first show that h defined in 7 is norm and Ag which is defined the below is convex set.
A (+][()) 1).
(19)
Proposition 1 (Convexity). Ap defined in above is concer set and h defined in 7 is norm.
Lipschitz Property
The Lipschitz property can be employed to show the boundedness networks over typical sets that are derived from the presented regularization scheme. The Lipschitz property of neural networks on Ag can be stated as follows.
Theorem (Lipschitz property on Ag). if E An and activation functions are aLiP Lipchitzz then we get
Ig0 Orlha s 2(alAp)" " V.|2
(M)+ 12 -T]F;
11
(20)
30

---

Published in Transactions on Machine Learning Research (08/2025)
where we set M Mc max |G' 1-11- Go is matrix induced by G and L is the number of hidden J
layers. We can rewrite (20 as follows
Dn 9rlaa S Ctipl r
where we define
CLip := ( ( V. ( +M++)))²
The above theorem results in boundedness An. This property proves useful for bounding the quantiles of empirical processes. Specifically, it helps demonstrate that the networks are Lipschitz continuous and bounded over lypical sels defined by our regularization stralegy (An). The following lemma formalizes this result.
Theorem (Roundedness on A ). The ((go E As). Ila) bounded.
Dudley Integral
To analyze the complexity characteristics, we introduce the covering number W(r, 0/ II) and define the corresponding entropy as
H(r, ..1 log. (f,
Where (r,, / I-D) is the covering number, r E (0, 0o), and |. is a norm on an space o/ ( (Vaart & Wellner 1996) Page 98). These quantities are used to define a complexity measure for class of neural networks 9n = {gn: E An).
We also define the Dudley integration of the collection of networks Gn == {ga Q An by
J ( = J H (r. (ln) dr,
for 8,0 o0) (fyan de Geer 2000 Section 3.3). Dudley integration be used to bound the complexity
of the corresponding neural network class. Now we try to provide the elements which are needed to find Xg
in Theorem 2 of (Taher et al. 2021). For this purpose, we apply Lemma 10 from (Taheri et al. 2021) which enables us to bound entropies over the parameter space rather than the network space. Moreover, we think of An as a set in R²" and we obtain
H (r, I. S H
Ab. 1F
VI +1)=
|00
= ((TT + 1)) |
Ao c RP 1...1-122
We also need the below auxiliary Lemma which relates Ag and the sphheee Lemma 2 (Relation between An and 1 spsphe of radius b). We have Ag Bı(b), where (b) is the 4 sphere of radius b.
Bounding the Dudley entropy integral is a key step toward establishing an upper bound on the Rademacher complexity. In the next step, we provide an upper bound for the Dudley integration.
Lemma 3 (Entropy and Dudley integration upper bound). Assume that the activation functions f RD T are OLip MLip-Lipschit, continuous with respect to the Euclidean norms on their input and output spaces. Then, il holds for every r E (0, oc) and 8, (0, oo) which salisfy < Sa that
H
Ab C .1 1- S
((LLI)) L (2 (M + =))² |=|n
246 GT
ePr2
log(
V 2e),
,2
cb2c2.p
31

---

Published in Transactions on Machine Learning Research (08/2025)
and
J (6, 55(M + +1jcTip log EP EP(M M ) v 2e
SoR
log
where
R' 2(arap) v Vi (T.) (7)((M (M+ +) ++1
and
R = max {R', 1..
B
Appendix: Proofs
In this part, provide proof for the main results of the paper.
Proof of Proposition 1
Proof. Because for θ,, O2 An we get
h + (1-a)0²)
[G(aO) + (1 - (1= + v/lao, + (1-1 a)
where θ = G +0 and 0₂ = =, + 02
Using triangular inequality, obtain
+ (1- 0)
=0 a)v|@2l
ab[O,] + +1-
For the second part of proposition, we have
[[a0] |G + v|a0] |a| b(0].
Moreover,
b|O] = 0 + |GO| | v/|0||
1. |G0|| 0
|0|1 0
from (1) we obtain that GO 0. Since G G)r we obtain that θ 0. Therefore, (II) results that
θ 0. Furthermore, by convexity we obtain
θ + O2
1
Og] =2h
2
2h
+ 2h
2
2
b
+ b
32

---

Published in Transactions on Machine Learning Research (08/2025)
Proof of Theorem 3
Proof. Similar to Taheri et al., 2021 we employ Proposition 6 of (Taheri et al. 2021 to prove Theorem
Compared with Taheri et al. 2021 we can use the below equation.
(
II
(IW* L2 =
(I
max LE LE)
jE!
591
3
Our goal is to find an upper bound for Σ (I V [V in terms of h. Considering Equation
17] we get
W W W ker(G).
As a result, employing 1. triangle inequality, 2. invertible matrix Go induced from G 3. norm property
Definition 3 4. factorizing the largest. norm value and using Equation 18
using b[S] s yield
Σ W = Σ W + W
$ I 11 1
G + [
(21)
I "I1 1 1+ [
s max & 11-1b θ) |6
of
s max
|1-1 + |+|,,
where the last inequality we restricted ourselves to the Ap. Also since 6|[]] S 1 we obtain
0101 $
Then we can write inequality 21
88 follows
Σ I
(22)
where M is defined in Equation
15
As a result equivalent to Lemma of TThhri et al. 2021 we get
(IW')22 12 1 < Σ (IW*)2 12 +1*122)
<
([W>), + 1V'1))
j=0
2
< 2M + w'
and therefore
33

---

Published in Transactions on Machine Learning Research (08/2025)
II
(IW* v 12)
IE lE(
je jE(U....L
1922
1
(w (V
7=0 1ut
1 M + )) +
$
According to Lemma
Taheri et al. 2021 for θ, l' E Ab we get
lBe lr S 2(arap) " L|
II
(IW' V V Flm.
je [E 141
Therefore, 12 P E Ab we obtain
|g grln I 2(0Lip) VI|x (Z [
where we define,
CLip = 2(LLi)) VV (M )
Proof of Theorem
Proof. Because according to Theorem
3
(Z(M++)) - 1
Igg - golln I V(Lip)=VIIlal
(23)
and according to 22 we find that
|0 1 Σ Iw'l $ M 1 ".
(24)
Proof of Lemma 2
Proof. Again employing 1. triangle inequality, invertible matrix G induced from G 3. norm property
in Definition 3 4. factorizing the largest norm value, applying this definition that = max (M, 1). and
using Equation
18 yield
34

---

Published in Transactions on Machine Learning Research (08/2025)
WEA - W
<
W 1 1
1=1
L Σ
|G G,
W 11+111
1=1
GGj
G W 1++1">/
bb
G, W
Proof of Lemma 3
Proof. By employing Lemma and using the fact that If al C B then N(E, M, s B, 1-11), get
N(c,A,,1llD 11122) SN(c (b) C R 1R,,)))))))
(25)
= =NG (1) C R 1.1)
Then using 1. the dcfinition of the entropy 2. Equation 25 3. definition of Crip obtain
H
A C CRP. .1.12
(M M + b)) | 2(aLip) (2
= Wo
r
An c 1 112
2((tap) LL V (2 (Z(M 1)) |
< log
B1) c RR,
2b(aLap) (3, (M )))
|00
log.N
(1) c R ,1 12
2be1ap
From Taheri et al. 2021
we obtain
H (22, Bi(1) C 11.11-12)
log.N (v2 Bi(1 RP,
S log(2ePp² V 2e), 112
where ;= 2by2cup
This yields
H
AB CRP.I1 12
2(atip) L /T. M + 1)) |00
<
(V214 (1) P 1.11
2462c2 oT
eP
log(
2
,2
4b2c2
35

---

Published in Transactions on Machine Learning Research (08/2025)
We need to find constant R [0, oo) so that supneA, Igah R. employing Equations 23 and 24 find that
R' == ((arap) 2 ( ) L+1
and
R max R 1}.
Therefore, for all I' E An it holds that A (r, Go I In) = 1 for all r > R and, consequently, H (r. 9b. |I (x) = 0 for all r > R. Thus we assume that r S R, as a result
/
0,
Hà (r. 9b. 1. 12) dr
50
24b² Lip
eP
log(
V 20))
dr
x2
462 Lip
ePR
1
- 5belin
log(
V 2e)
dr
c Tii
$ r
cPR
8
= 5bCLiP
log(
2e) log
V
cf
((2((Li))'' ( 1) 1)
=
V
( (2 + 4
S
5borip log
eP(M + 1)²
S 8
= 5belip V log(
V 2e log
6
S 5(M + 1)crip log(eP(M + + v 26) log
S 5
Proof of the main result of Section
(Theorem
1
Proof. We recall that
1oa - grln S 2(aLip) 2 (z +)) t.
12-Tlh
(t) (M )
lgalnn s 2(01ip) V VL|=
J (0, 0 An) s 5(M + 1) log EP cP(M + ) v 2e x
80
log
As we see in the Section 4.2.1 in practice M is parameter whose value is less than one and close to zero. We set v 2 I-M As a result we see that the above upper bound can be simplified as follows
8 R
(6 $ 5(M + 1) /og (2P) log
8
36

---

Published in Transactions on Machine Learning Research (08/2025)
H'' = =((aan) 2
which is the first ingredient need to employ corollary 8.3 of (van de Geer 2000 In order to apply this corollary we also need to choose parameters ó, E o0) so that
8 < RR,
and
Vn
Osub
(J(8,o,Ap) v R).
Now we will try to find proper values for them. For simplicity we define:
m),, L, P) = 5(M M + 1)elip v log (2P).
Also we set
8 == 4d R x log (2n)
Vn
where f is a parameter related to M, L, P that we will find later.
In this step we first find an upper bound for S that we need in our computations. We define 28/RV V7
1. by definition of a and 8, 2. assuming 4 20auh 2 7/5 we get
80gub Rx lon(2n) vi
8R
V v27)
8
R
6
40sub R x (lp(22))
vn
(26)
16 V ( 2 2/27v/
16/n
16
16
fasub log(2n)
log(2n)
As result, using 1. Equation 26 2. the fact that log is an increasing function and R 2 1:
Osub
(RVJ.
vnaa [RV r (M, L, P) log 8c
8
4a R x f(M, L, P) log 2m)
Vā [1 TMMM, L, P) log
4fM, L,, P) (2n)
(M, L, P) 2f(MLL,P)
=v
4f Pog
If we assume that n is enough large and by selecting constant f= 2 we get
0gub (JV R)
S nn,
which is satisfying the second condition as desired.
37

---

Published in Transactions on Machine Learning Research (08/2025)
Therefore, using 1. Lemma 11 of Taheri et al. P (co DD) 2 - a and P cc)
2021
(where : o²), 2. the fact that P(C n DD results
c D P (D)) we obtain
2 P
n Σ
})
P
sup READ
(a(2;) )uel 26 20)n
u S 0²
==1
S Osube (9)
n (0(2)))ll s 0)
P
sup REAÁ
(27)
-ne? Osub (Boub RJ2
e
Σ sa( = 20(( 21-
P
sup REAB
where in the last line we used the fact that
Asuble 77
log
2 vn
5(M + 1)CLip v log (2P) log (n)
asub x
2
/n
and
naa sub R² 25(M+1)² log(2P)
log(2n) vn
-n882
(guh 12)2
(asub
25 (M + 1)3² Lip log (2P) (log(2n))².
1
Equation 27 states that A,,1 I 25 for t 1/n. Then the main result can be implied from Theorem 2 of Taheri et al. 2021 by setting À 2 26 20 (l0(22))
Proof of Corollary 1
Proof. We know that
G-1 Im( -
for G 0 we have (m) 0 and k 0 (as the ker(O) contains all the space). Therefore, G-1 is also equal to zero. As result Mo 0 and Theorem holds for every 2 1.
Proof of Upper Bound for Parameter M (Lemma 1
To prove this lemma we first prove Corollary G This lemma is a direct result when we have just one partition.
Proof. We know that A MMpzixnn + Mixxn where Mpje1xp, = ker(A)) + ker(A)) (0 denotes the direct sum) and Ij = E. (1 We also have
kcr A (W': consist of same rows in each partition).
In fact.
38

---

Published in Transactions on Machine Learning Research (08/2025)
A Ber(A')= @ ker(A - Im(A') C Mi,xpy>
and by definition A ker(A)) 4 Im(A)) is the restriction of A to the subspace kcr(A Therefore, it is invertible and we obtain
A Im(A') + ker (er(A
We know that groups provide partition for rows therefore A + Pi where A is the matrix associated to each group. We also know
A Miyxp, - ker (
by the geometric interpretation of A (pseudo-inversee of A)we know that it is an exlension of A A Im (A*) + (t)( We need below auxiliary Lemma for the rest of computations.
Lemma 4 (Equivalence of norm two spaces). 11 of A same the || 11-1 of A' linear map from MP Xp, to M,, xp,
linear map from R Rb is the
Proof. Let D E MMs psixxp; and let denote its columns by d1, d1...., ... Therefore, A'(D) [['dd,.... d'p By 28 29
h-n1 of A as linear map Mp Xps to Mixxp, is equal to || -ll1-1 of A as a linear map R
from RPS+1
Also by definition
A x/11 |0||
1-1
sup #EI
and
AAi x/1 |x||1
1A 11-11
sup a MEMi, xps
Since the sup is laking over larger set get
11-1 I 14 |1--
Note that A is a linear map from the linear space Mpstixpy to the linear space Mixxps- If D Mişxp; then:A (D) (A') (orthogonal projection of D on Im(A)) which shows ker A is an extension of A Because of special structure of matrix A we can see that
AP"
A
BP
therefore:
|A h-11
VP.
On the other hand, if
= B + B ....
(28)
then
B 111-1
sup
B 11
(29)
39

---

Published in Transactions on Machine Learning Research (08/2025)
As result
M sup |A'
11-11 I sup sup | 11-11
2
sup sup VPr.s |3/
Proof of Prediction Bound for Tensors and Boundedness of Effective Noise (Theorem 4
Proof. We have
- (X, - AM[A].
Adding a zero-valued term in the normss on both sides then gives us
(x, B) Y ( Xb
s Y (x, B) L + (X, A) Ab
The fact that |C| = (C, D) and expanding
|C D (C. C) + (B, + 2(C, (D) D. yields
|| [X 1). B), (x, B) 1
2 Y - X B) 2, (), (x, (), M+1
+ X
S Y - X (x (X, A)
2 2(Y - -x + Ab[A].
Then we can then derive the below inequality.
1X, - (X, S -X
+ 2(U, (X, B) (X, A)
(30)
[ + AA[A].
This inequality separates the prediction error of the estimator from other parts of the problem. We now try to control the inner product term the right-hand side. Employing the properties of the inner product we can write this inner product as follows.
(U, (X, - -X (U, ( A)2)M41
(XX, U)1. - A) U), B) B) ((X, U31, M+)
Then Holder's inequality gives
()x,),, S U), |B|,
and
(x, U)1 A)M+1 |[X, 1/ |A
By replacing them in Equation 30
we obtain
|X, B) - X -
(31)
+ (X, 2 11 + A6[A].
40

---

Published in Transactions on Machine Learning Research (08/2025)
Similar to the previous case know that the regularizer is norm. Employing the incquality between norms we have ||A]]] I Ch(A] where C is a constant. By replacing in 131 we get
(x, (X, BX (x
(x, 1b 21,X, U1) Xh + Ab[A].
Assuming X) & À we obtain
1), B -XX 22b[A]-
Since A was selected arbitrarily, we get
B), 1X, s inf (1(X, (X, A) + 216[A]}
We now show boundedness of effective noise in the tensors. By definition of the sup-norm, it holds that
2 11
((X, U)))pp.... Pr.0.. AM
EI P Pr) (( Q..- E (1..... Qm1
Therefore, the complement of the event as follows
{}+ 2 (X, ) (21(X,U)11 Ae)
P1
QM
U
U
22 ((X, Uh1 Punrir aM
> >
P1=1
9===1
Applying the union bound, yields
IP [2] (X, U)1 > 1115
P
QM
Σ P1=1
Σ
P f22 (1x, U)
> No).
4M =1
and we obtain
P(2 (x, U/1o0 > Ax) < P1 ........1.... .. .
P P(2 ((X, Un)) ().))..... Pr.... AM Ar.
sup
P1E(1. .l......... PI)--LE( (I..... Pz) E (l......... 21 J.-MM El Qar..
We define
sup El....... Q.) (x.
2 ((xxX)) 9.. .MM -1.- AM
/NN
41

---

Published in Transactions on Machine Learning Research (08/2025)
then we get
P (2) 1)1
$ PI.... ... ..PrQQ ... Qu
sup
PIe (1.... Pt) P.) PI......... PL} (ve()...,Qu Qxx
P {22 ((X, 1) pl PL RM
Ae
..... P.Q1...... Q.
sup
PIE Pi ). (l.... eEffl.... Q1 p.ggar E Qur
((X, p..
0 AM
d
20 N
20 No
Pi P....PE.i ... QM
sup
PIE 1 PP E (I....PP.. mE Q1) J. E 1 Qxx }
((X, U11 PI....
d 20
P
((X,, X) 101........ -M
1))))
where (10,8)1) p1. P1.0 pp. aM
1).
((x,8))) war.u1
Because, U Un DL o2 and X. .m." aas/ ((x, X) 1) ax 0 AM belongs to the unit sphere
(Lemma 3.1 of (Xu & Zhang 2019
Therefore, we can employ the tail bound
P[[a| 5e for all 2 0.
Combining this tail bound evaluated at (20 we get
P (12(X,U)ll A1) P1 PLQ1 .Qre (Zniir))
As result
P( 2 21(X,U))
=1 [2] 10 Ao3
2 1 - P. ...PPQ Que (1--1100 (Zo//==) )/12
C Implementation
This section presents the implementation of our main algorithm (Algorithm 3
import numpy as
import matplatlib.pypl.. as plt import time import math
def encode_columns by_ Eiirst_eeeeent (matrix):
num_rows len (matrix)
num_cols (matrix(01)
encoded_matrix [[O for range (num_cols) for range (num_rows) range (nun_ 8):
is_comp1 matrix [0] [col]
42

---

Published in Transactions on Machine Learning Research (08/2025)
in zange rowa]:
value matrix [row] [col]
encoded_matrix [col] (value 1) if is_complement else mp.array
def compress_matzze wwt.rr.bby_.rrssteereeee First_ (matrix):
18
firat_row [
return [[val, (val 1) $ 2] for val in first_row]
def
encode_xoke cece__oww___pp____ by firsttttt_____ _elment (matrix):
for row in matrix:
if row[0] == 1:
for i in range (1en (row)); row row[i] = (row[i] 11 2 return matrix
def Compress_matrix by_ firat_col [zow[0] rst_columm (matrix):
in matrix]
[[val, (val 1)
in first_col]
compressed wtapeee___triixmiiiiiie (t_ index_matrix_M, U_list_B, Index_matrie nun_products len (inndxx_matriix
34
num rows len (enddee__matziixAAl num_cols (en(nnex_matrrix23
35
36
compressed_prodieee for i in (angepuuuurood
product compute__ouree__prooicc.. (t_ (t_list_A(i), V_list_B(il)
result_matriz zp. (numm_rows for k in range (mm__ppuucs)) nun_rows_sub = len sanlccmmresseed_rooccttaa
ior num_cols))
num_cols_sub len penpppreseed_rroustttee [0])
for in range (num_rows_suu)) rok_indices [idx for idx, in enumerate (index_matrix_Alg k]) if val for in range (num_cols_stbb) indices [idx for idx, in (Lndex_matrix result_matrix (np. (row_indices, col_indices)] compressed products (1)
val
result_matrix
def compute_outed prupppee_uterrppoouctt
product (vector_a, vecter_b).
82
return [[vecter_a[A] vector for j in range/len vector_a)]]
(len (vestor_b))) for in range (len
def standard httpr/aaaaaaaattiaattaaaoopoo (matiiiiiiiiiyyaaaaaa.aa matrix reault = np. zeros (len (natrix_a), len (matrixxx(())) for i in range (len (mat=ix_s)):
for i in range (1en (mat=ix_b(01))):
for k in range (len (matrix_b)):
result +- matrix_a[i] return result
mkkr
def compute compute_frobenius tamieeniuussoommmsriiiii
math.sgrt. aemmeeement for matrix for element zew)
fast_ []
Saandadd=imme []
matrix_sizes [30, 100, 200, 200]
89
print ("Comparing fast (compressed) va standard binary matrix mitdplication:"") print ("=" 60)
71
for size in matrix_sizes:
matrix_A = randon, randint (2, si ze= ziizetsi (size, sizel)
matrix_B = wpp.ranon.ranninnt.
randint (2, si zizztsizo sizel) start_time = time.time
encoded_A
(matrix_A)
encoded_D B encode_roves first
srirst_element element (matrix_)) (ww.srrttttwtttttttssss
compressed A compress_matrise ftrttstttoorrrr_troeeeere row (mstrix_.)
43

---

Published in Transactions on Machine Learning Research (08/2025)
compressed_ compress_matzine wttsttyyffrrttoooooooooe (matrix_B)
1ast_result wwmgressed__mtzix_mmiiiee (compressed encoded compressed encoded
fast_time time.timm start_tine
fast_ttmee.agpenddfaas append (fast_timm
start_ time.time
Standard_resuil standard
m)trix_B)
standard_:ime time,tine start_time
error_matrix - fast_ standard_result
error_norm
print (f"Marix size (size) -> Frobenius norm error: (e:ror_norm:.2f)
(tt.iiguueeffigsizee:11 61)
glt.plot (matrix_sizes, fast_times, marker-"o', label-'Past (Compressed)+ color="blue"
Flt.plot (matrixsizees, standard_times, marker='s', label=' Standardd, color= red") [it.xlaabal""Matrii size (r ()
E.t.ylabel(''img (seconda)')
elt.title(''lnncy Matrix Multiplication Performance")
glt.legend()
(lt.grid(True)
glt.tight_
100
slt.savefig ("w.iiimmmmttiiiiiaa pngg)
Listing 1: Cardinality sparsity vs Standard Binary Matrix Multiplication
import numpy
import pestpletiib.pypoee plt
import time import math
def custom unstom (zow)
unique_el seggue_elementte first_indices
counts = []
10
for idx, value in enumerate if value not in seeuuueeeeeeettee anique_slement s.mments.appenet append (value)
11
12
13
prrst_indices.appent (idx) counts.appendil.
else:
el element_indes mnique_element index (value)
@ounts@@lement_nnee +=
return arrar (uniqueeeeemmnttss
ap.array (counts)
def encode_firse cocood__frrst__triia_ (matrix_A);
num_rows len (matrix_A)
num_cols (matrix_A(0])
mncoding_matrix zerosr (num_rows, 1))
unique_blocks
flat_indices []
for col in range (mun_cols):
27
unique_vals, first_indices
custom custom_unique (macrix col))
28
block_valued (matrix_A[inde (index, col) for index in sorted(((rss__ndddces)
29
(bllok___
30
for block_idx, valus in enumerate (b) ock_values):
matching indices [idx for idx, val in enumerate mmttrix_Ali, coll) if val -- value] wwtp:/maaiiiiiii (matciiaaaectiggniiee.aa indices, :] block_idx
tlat_indices - np. append (feen (fdet_indiees, index_natrix = np reshape (flat_ (num_cols, num_rows))
sesodiing_matrll
return unique_blocks, index_matrix
def encode__eeeeeeeeee wtwp__secoodd_ar.ii...e (matrix.3):
num_rows len (matrix_B)
num_cols (matrix_B[O])
incoding_matrix zeroa nam_cols))
unique_blocks
Flat_indices
44

---

Published in Transactions on Machine Learning Research (08/2025)
in zange (owa] unigue_vals, first_indices, block_valuee [matrix_B[ret [row, idx] idx (frted
unique Matrix_B(xoH, :])
(oo for block_idx, value in enumerate (block_valuesi)
indices [idx for idx, vai in enumerate (matrix Bizow, :1) if val value] incoding_matrixlli matching_indicee indices] block_idx
49
flat_indices = np. ap.apd (flat_ididd index_natrix = np. reshape (flat_indices, (num_rowa, num_cols))
encoding_matrix)
return unique_blocks, index_matein
def compure__uter_producet (vector_a, vector_b):
rows = (vectoror_a)
cols len (vector_b)
result [[0] cols for in range (rows)
for in range (rows):
for in range (cols):
reault(i]li) vector_a(1] vector_b[]]
result
index_M, blocks_B, index_B)
num_blocks [en
num (indeex_A(O))
num_cols 1en (index__B(O)
compressed_produeee = for k in range (tun__hokkloc outer cowpute_outeeeeeccoeeeeeoooooooe wttpatte__ueeeerrddeeeee (blocks_A(k)) blocka_B(k))
result_matrix np.zeros (num_rows, num_cols))
for block in range (num_blocks): block_rows = len (ccoppeeseed_rodusttttbbbbb (block))
block_eols len (copppressee__rooatttttsss (blccc [bleck] [0]]
for in range (block_raxes rok_indices [idx for idx, in enumerate (index_A [block,, :1) if val for in range (block_cols)) col indices [idx for idx, in enumerate (index (block, :1) val j] result_matrix (np.xx/pp.... (ros_indices, col sol_indices]] compressed_ [block] (1) (3)
result_matrix
maaadiid_matrixxmmm www.iplyllpplyyyyo... (matrix_A, matrix reault ap. zeros (len (matrix_A) len (matrix.Bl0))
for 1 in range (len (matsix_A)):
for j in range (len (matrix_B((])))
for k in range (len (matzix_B)):
(eault(i) [i] += matrix_A[i] makr
return result
def Srobenius_nores (matsix) return math. sert (sum (element for row in matrix for element in rew)
fast_times []
standard_times iimes
matrix_size [200, 250]
value blas www.raann.ooom..mmm. (loc-3, scale-2)
for size matrix_sizes
matrix_A np. randan.. matrix_B ap start_ time. Lime.time ()
aanadooa.randint (10, size=(size, size)) value_bias
100
(10, size-(size, size)) value_bias
101
102
compressed_A, A, index_A
103
compressed_B, index_B aroode_secoom_maatriiiieeee (matrix__ seault_compresee compressed_mattix_mmlle (compressed_A, index compressed_B, index_3)
104
105
el laased_fast time. start_time sintstimes.oo (
106
107
expscted_result dot (matrix_A, natrix_B)
arror_matris expected_resuff zesult compreseed
45

---

Published in Transactions on Machine Learning Research (08/2025)
109
print (f"Matrix size (size)x(size) Frobenius norm error: (frober.us hozm matrix) :.2f)")
110
start_time time.time
111
result_sstanarrd standard_matrix_miitiii mit.trri...illllyymmmma..i... matrix_3)
112
standad (ime)time() start_time
113
standard apwpdddeeeaseeddsaaaaae..
114
115
# Plotting
116
pit.figure Cet.fiigue(ffggeeiee (figsize= (10, 6))
117
ax pit.gea()
118
ax. 2x._et_faccecloor(1100001
119
(matrii fast_times, color='blue', marker='o', label=' multiplication'
120
alt.ppoot standard_tines, color=' red', marker='s', label=' Standard miltiplication')
('Tixe (seconds)
Flt.ylabel ('Matrix size (r m)
Nlt.tittla Matrix Multiplicath Performance")
124
git.grid(True,, linestyle='==', 1inexidth-0.5)
125
eet.LegenddLee-uppet left')
128
elt.tight__ayoutt() plt.show()
127
128
129
print ("fast_ ffaes_times:", fast_ (ast_timea) print "sstaddard_tlmess" standard_times)
130
Listing 2: Cardinality Sparsity for Non-Binary Matrix Multiplication
46


Reviews:
[{"review_id": "FQ3E3XqxoL", "reviewer": "Reviewer_Y4Zd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++:\n1.  The concept of cardinality sparsity is original and meaningful, with clear motivation and distinct advantages over traditional definitions of sparsity.\n2. The paper extends previous statistical frameworks to accommodate the new notion of sparsity, particularly through generalizations in DNNs and tensor regression.\n3.  The results on accelerating matrix-matrix multiplications are impressive and practical.\n4. : The empirical section is comprehensive, covering use cases in deep learning, matrix factorization, and classification tasks.\n5. The approach could benefit low-resource machine learning and efficient model deployment, particularly in edge computing or embedded systems.\n\n---:\n1. While mathematically sound, the theoretical sections are somewhat dense and could benefit from clearer exposition, more intuitive explanation, and illustrative examples.\n2. The manuscript could better emphasize how cardinality—as opposed to classical sparsity—drives performance gains theoretically. While this is implicit, a more explicit contrast would help readers appreciate the contribution more directly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel concept of cardinality sparsity, defining sparsity in terms of the number of unique values in a tensor, rather than the number of non-zero elements. The authors demonstrate that this formulation enables both statistical and computational advantages in various machine learning applications, including deep learning and tensor regression.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major:\n1. Please improve the readability of the theoretical sections by simplifying the exposition, introducing more intuitive explanations, and reducing notational complexity where feasible.\n2. While the paper is strong overall, the empirical validation in Section 6.3.1 remains relatively limited. The authors are encouraged to strengthen this section by evaluating additional aspects such as varying neural network architectures, different training dataset sizes, and broader performance metrics beyond training accuracy—e.g., test accuracy, generalization performance, stability, and sensitivity to regularization. A richer set of experiments would substantially enhance the credibility of the proposed sparsity approach for deep learning applications.\n\nMinor:\n3. The citation formatting contains inconsistencies. For example, the citation “Bowen and Wang” is missing the publication year.\n\n4.  Section 3, first paragraph, line 3, there is a missing space between “sparsity.” and “This”.", "broader_impact_concerns": "I did not identify any significant ethical concerns in the current submission. The broader impact of this work appears positive: it enables more efficient computation and memory usage, which is desirable for environmentally friendly machine learning. There is no indication of misuse or safety issues given the scope of the methods.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DBdc32KkhB", "reviewer": "Reviewer_JzCx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The concept of cardinality sparsity is genuinely novel and well-motivated as unlike traditional sparsity, which requires many zero values, this framework can capture structure in data where values repeat frequently but aren't necessarily zero - this seems practically relevant for many real-world datasets. The mathematical formulation is rigorous. The authors provide formal definitions for cardinality sparsity of tensors using mode-n fibers, and the connection to difference tensors provides a clean theoretical framework for analysis.\n\nThe paper demonstrates impressive breadth, spanning matrix algorithms, statistical learning theory, and practical applications. The statistical guarantees (Theorems 1 and 2) appear technically sound and show that cardinality sparsity can achieve similar benefits to traditional sparsity without forcing parameters toward zero. The experimental results show clear computational benefits, particularly for binary matrix multiplication where they claim substantial improvements over state-of-the-art methods.\n\nWeaknesses: \n1. The matrix multiplication algorithm has a critical flaw that the authors don't adequately address. They claim O(mPn) complexity, but the \"decompression\" step (reconstructing the full result from the compressed computation) actually requires examining every entry of the output matrix, which is inherently O(MN). This means their algorithm can't actually be faster than O(MN + mPn), and for many practical cases where m and n aren't dramatically smaller than M and N, this provides no advantage. Moreover, the Cartesian product operation in Algorithm 2 (I^(1) ×_c J_(1)) is computationally expensive. The authors claim it's \"just concatenation\" but it's actually a mapping operation that requires O(MN) work to properly reconstruct the result matrix, again negating the claimed speedup.\n\n2. The assumption that matrix structure is \"known\" is not just strong—it's often impossible in practice at least to my knowledge. Consider:\n2.1 In neural networks, while input data might have cardinality sparsity, weight matrices typically don't (and shouldn't, as this would severely limit expressiveness)\n2.2 For iterative algorithms, matrices change at each step, requiring constant recomputation of the structure\n2.3 The authors provide no analysis of how often this assumption holds in real applications\n\n3. I guess even just a quick search on the internet, i see that the baselines are not very strong as they are now more specialized boolean matrix libraries that can achieve O(n³/(log n)^1.5) or better [https://www.cs.princeton.edu/~hy2/files/cbmm.pdf][Huacheng Yu,\nAn improved combinatorial algorithm for Boolean matrix multiplication,]. Also the authors they provide no analysis of when their method performs poorly or fails entirely.\n\n4. How exactly is the \"preprocessing\" done efficiently? This is handwaved and there is no code availability: Despite claiming practical advantages, no implementation or at least pseudocode is provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel concept called \"cardinality sparsity,\" which defines tensors as sparse when they contain only a small number of unique values, rather than the traditional notion of sparsity based on zero-valued entries. The authors at least to me make several contributions: (1) they formalize the mathematical definition of cardinality sparsity, (2) develop matrix multiplication algorithms that exploit this structure to achieve computational speedups, (3) establish statistical guarantees for regularizers that encourage cardinality sparsity in deep learning and tensor regression, and (4) demonstrate applications across multiple domains.\n\nI think the core idea is intriguing and potentially valuable, but I have some concerns about both the theoretical claims and experimental validation that I'll detail below.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some changes, again please look at the weaknesses above, but reemphasizing these points.\n\n1. Revise complexity analysis: Include preprocessing costs in the complexity analysis or clearly state when the \"known structure\" assumption is reasonable. For applications where matrices change frequently, the preprocessing overhead could dominate any computational savings \n2. Strengthen experimental evaluation: Compare against modern state-of-the-art algorithms for Boolean matrix multiplication, not just classical methods. \n3. Fix the memory claim analysis:  I feel like Table 2 memory comparisons are incomplete as you need to include storage costs for encoding matrices I and compressed matrices C_W, C_V and ccount for integer index overhead and pointer structures\n4. Also could you include the pseudocode? \n5. More ablations by varying cardinality degree, and ehavior with noisy data where values are approximately but not exactly repeated. Lastly would be good to do some performance degradation analysis as problem size scales!\n\nOverall again I like this paper and direction: ) but more details would be great to include and more real world examples!", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gqta4nE8nx", "reviewer": "Reviewer_WTA4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. Cardinality sparsity is an elementary but useful metric to capture the complexity of matrix / tensor operations. It is helpful to design algorithm that utilizes cardinality sparsity even beyond matrix multiplication. In particular, many numerical linear algebra algorithms have their runtime depend on the number of nonzero entries in a matrix / tensor. It would be interesting to design NLA algorithms when the matrix is dense but cardinality sparsity is small, and the runtime depends on the cardinality sparsity.\n\n2. The theoretical analysis of neural network and tensor regression is actually more general than accounting for cardinality sparsity, it gives bound on a class of regularizers. It would be interesting to find more applications of the meta theorem obtained in this paper.\n\nWeaknesses:\n\n1. The matrix multiplication algorithm for cardinality sparsity is very elementary. When the inner dimension of the matrices is small, then one could simply perform outer product over unique elements, then map them back to corresponding entries. When the inner dimension is large, one then use the unique elements and index set of the rows and do standard inner product by avoiding unnecessary multiplications. Once the concept of cardinality sparsity is spelled out, these algorithms become quite obvious.\n\n2. The theory regarding neural network and tensor regression is nice, but they are quite general with cardinality sparsity is just a special case. The theme of the paper shifts away quite significantly from cardinality sparsity when discussing this part. I do believe the results obtained here are interesting, but it might be worth thinking how to unify them better with cardinality sparsity.\n\n3. Regarding experiments, while the results show that algorithm based on cardinality sparsity is better than standard algorithm or Strassen, I'm unsure about the practical implication here. Matrix multiplication is a highly optimized operation and modern GPU architectures are designed for them. If one compares the proposed cardinality sparsity based algorithm with matrix multiplication on GPU, it's unclear which one is faster, at least authors haven't specified on what platform the experiments are performed. Of course, it's not fair to compare a proposed algorithm with GPU-optimized matrix multiplication, so I think a fair comparison would be that, if one casts the cardinality sparsity based algorithm into standard matrix multiplication, how many matrix multiplications of what size matrices are needed. I do understand this might be challenging, as for cardinality sparsity based algorithm the matrix multiplications are for matrices of non-uniform size. I think, at least, the authors should clarify what platforms the experiments are performed, how do they implement the standard algorithm (just sequential algorithm that computes inner product entry-by-entry without any parallelization?), etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies a parametrization of matrix or tensor which authors termed cardinality sparsity. Roughly, it measures the number of unique elements for each mode of the tensor. This is an intuitive measurement of sparsity, as when performing matrix multiplication, the number of necessary multiplications is captured by this parameter. The two main contributions of this paper are: 1. matrix multiplication algorithm utilizing cardinality sparsity, 2. a theoretical analysis on the prediction guarantee of neural network and tensor regression. The theoretical framework accounts for a class of regularizers including the standard $\\ell_1$ regularizer and cardinality sparsity is a special case of them. Empirical evidence shows that matrix multiplication based on cardinality is faster than the standard algorithm and Strassen. It also improves the parameter count for training 2-layer ReLU network.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major:\n\n1. See my comments in weakness regarding experiments.\n\n2. I think the authors should try to reorganize and emphasize on the cardinality sparsity part for neural network and tensor regression part. Currently, it feels like the theory part is a bit redundant and not very coherent to the matrix multiplication part. I'm not super sure what's the best course of action here, maybe start with cardinality sparsity as a regularizer, then generalize to other settings instead of the other way around?\n\n3. On the other hand, I feel it might be more coherent if authors could strengthen the results regarding matrix multiplication by providing some downstream applications that can be solved in cardinality sparsity time, such as regression, low-rank approximation and other numerical linear algebra tasks. One candidate would be conjugate gradient -- the algorithm is essentially matrix-vector multiplication, so I guess you could implement each iteration in cardinality sparsity time?\n\n\nMinor:\n\n1. The citation style mixed with not differentiating \\citet and \\citep makes reading a bit challenging, authors could either choose another citation style (say number, or alphabetical) that does not require differentiating between the two, or fix the mix use of \\citet and \\citep. Also many references miss years, making the already hard to parse mixing of citation even harder to read.\n\n2. References for fast matrix multiplication are outdated, consider citing [Duan, Wu and Zhou, FOCS'23], [Williams, Xu, Xu and Zhou, SODA'24] and [Alman, Duan, Williams, Xu, Xu and Zhou, SODA'25].\n\n3. Section 3, second line: \"In order to sake completeness\" should be \"for the sake of completeness\".", "broader_impact_concerns": "This paper focuses on a new metric to measure the sparsity of matrix, tensor and more general machine learning models, it does not have any direct negative ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)