Low separation rank in tensor-variate generalized linear models
Creators: name: Taki, Batoul Ahmad
Publisher: name: No Publisher Supplied
Year: 2024

description: Since many real-world measurements are inherently multidimensional – involving multiple dimensions or modes – tensors, which extend vectors and matrices to multiple dimensions, are crucial in handling such data. Recent research in machine learning and statistics has shown that the performance of traditional vector- based approaches in solving regression problems often falls short in high-dimensional and small sample regimes. Many tensor-structured machine learning problems are high-dimensional, exemplifying the need for flexible tensor decompositions to tune model complexity to match the available data. Specifically, when the data is tensor structured, imposing tensor structures on the coefficient tensor in regression problems can lead to a more reliable parameter estimation and lower sample complexity compared to vector-based methods. The choice of tensor structure to be imposed affects the representation power and generalizability of the regression model parameter. Compact tensor structures greatly reduce sample complexity yet may not capture more complex interactions in tensor data. Conversely, more flexible tensor structures may suffer from a larger variance and sample complexity. This dissertation investigates a new low-rank tensor model, called Low Separation Rank (LSR), in Generalized Linear Model (GLM) problems. The LSR model – which generalizes the well-known Tucker and CANDECOMP/PARAFAC (CP) models, and is a special case of the Block Tensor Decomposition (BTD) model – is imposed onto the coefficient tensor in the GLM problem as a means to enhance model representation power whilst maintaining an acceptable level of compactness. This dissertation explores both the theoretical and algorithmic aspects of parameter estimation in LSR tensor- structured GLMs (LSR-TGLMs). It is comprised of three parts. The first part of this dissertation establishes the learning theoretic framework of Tensor Generalized Linear Models (TGLMs), through the lens of LSR tensor-structured regularizers. To estimate the LSR- structured model parameter in such problems, a numerical method based on alternating minimization / block coordinate descent is developed. The computational advantage of such methods is highlighted by the fact that a larger non-convex problem can be divided into smaller convex subproblems. Numerical experiments on synthetic datasets demonstrate the efficacy of the proposed LSR tensor model for three regression types (linear, logistic and Poisson). Experiments on a collection of medical imaging datasets demonstrate the usefulness of the LSR model over other tensor models (Tucker and CP) on real, imbalanced data with limited available samples. The second part of the dissertation delves into the theoretical aspects of the LSR-TGLM problem. It ex- amines the fundamental limits on estimation error through minimax and excess risk bounds. These bounds are critical for understanding the sample complexity of the parameter estimation problem in the LSR-TGLM framework, and provide a benchmark for algorithm efficiency. The minimax lower bound on the estima- tion error is proportional to the intrinsic degrees of freedom in the LSR-TGLM problem, suggesting that its sample complexity may be significantly lower than that of vectorized GLMs. These bounds can also be specialized to CP and Tucker-structured GLMs, and their tightness is assessed numerically and through the derivation of sample complexity results (excess risk bound). Additionally, sufficient conditions for local identifiability of the LSR-structured tensor in the LSR-TGLM problem is discussed and results on asymp- totic consistency and normality are established. This involves proving new properties of LSR-structured tensors, particularly compactness of the set of LSR-structured tensors. The asymptotic consistency result shows that maximum likelihood estimates of the parameter estimation problem converge to the true values as sample size increases. The asymptotic normality result shows that the maximum likelihood estimates converge to a normal distribution. The third part of the dissertation extends the algorithmic framework of LSR-TGLMs to the federated learning setting. Federated learning enables collaborative analysis without centralizing data, which is par- ticularly relevant for biomedical imaging where data is often distributed across multiple sites. By balancing local update steps with communication rounds, the federated algorithm achieves performance comparable to centralized methods, thus enabling effective collaborative analysis without the need for data centralization.