Title: Coded Computing for Resilient Distributed Computing: A Learning-Theoretic Framework
Venue: NeurIPS 2024
Accepted: yes
Keywords: Coded Computing

Distributed Computing

Non-Parametric Regression

Smoothing Spline

Kernel Ridge Regression
Abstract: Coded computing has emerged as a promising framework for tackling significant challenges in large-scale distributed computing, including the presence of slow, faulty, or compromised servers. In this approach, each worker node processes a combination of the data, rather than the raw data itself. The final result then is decoded from the collective outputs of the worker nodes.  However, there is a significant gap between current coded computing approaches and the broader landscape of general distributed computing, particularly when it comes to machine learning workloads. To bridge this gap, we propose a novel foundation for coded computing, integrating the principles of learning theory, and developing a framework that seamlessly adapts with machine learning applications. 
In this framework, the objective is to find the encoder and decoder functions that minimize the loss function, defined as the mean squared error between the estimated and true values. Facilitating the search for the optimum decoding and functions, we show that the loss function can be upper-bounded by the summation of two terms: the generalization error of the decoding function and the training error of the encoding function. 
Focusing on 
the second-order Sobolev space, we then derive the optimal encoder and decoder. We show that in the proposed solution, the mean squared error of the estimation decays with the rate of $\mathcal{O}(S^3 N^{-3})$ and $\mathcal{O}(S^{\frac{8}{5}}N^{\frac{-3}{5}})$ in noiseless and noisy computation settings, respectively, where $N$ is the number of worker nodes with at most $S$ slow servers (stragglers). Finally, we evaluate the proposed scheme on inference tasks for various machine learning models and demonstrate that the proposed framework outperforms the state-of-the-art in terms of accuracy and rate of convergence.

Coded Computing for Resilient Distributed Computing: A Learning-Theoretic Framework

Abstract

Coded computing has emerged as a promising framework for tackling significant challenges in large-scale distributed computing, including the presence of slow, faulty, or compromised servers. In this approach, each worker node processes a combination of the data, rather than the raw data itself. The final result then is decoded from the collective outputs of the worker nodes. However, there is a significant gap between current coded computing approaches and the broader landscape of general distributed computing, particularly when it comes to machine learning workloads. To bridge this gap, we propose a novel foundation for coded computing, integrating the principles of learning theory, and developing a framework that seamlessly adapts with machine learning applications. In this framework, the objective is to find the encoder and decoder functions that minimize the loss function, defined as the mean squared error between the estimated and true values. Facilitating the search for the optimum decoding and functions, we show that the loss function can be upper-bounded by the summation of two terms: the generalization error of the decoding function and the training error of the encoding function. Focusing on the second-order Sobolev space, we then derive the optimal encoder and decoder. We show that in the proposed solution, the mean squared error of the estimation decays with the rate of O ( S 3 N -3 ) and O ( S 8 / 5 N -3 / 5 ) in noiseless and noisy computation settings, respectively, where N is the number of worker nodes with at most S slow servers (stragglers). Finally, we evaluate the proposed scheme on inference tasks for various machine learning models and demonstrate that the proposed framework outperforms the state-of-the-art in terms of accuracy and rate of convergence.

1 Introduction

The theory of coded computing has been developed to improve the reliability and security of largescale machine learning platforms, effectively tackling two major challenges: (1) the detrimental impact of slow workers (stragglers) on overall computation efficiency, and (2) the threat of faulty or malicious workers that can compromise data accuracy and integrity. These challenges have been well-documented in the literature, including the seminal work [1] from Google. For instance, [2] reported that in a sample set of 3000 matrix multiplication jobs on AWS Lambda, while the median job time was 40 seconds, approximately 5% of worker nodes took 100 seconds to respond, and two nodes took as long as 375 seconds. Furthermore, coded computing has also been instrumental in addressing privacy concerns , a crucial aspect of distributed computing systems [3-12].

The concept of coded computing has been motivated by the success of coding in communication over unreliable channels, where instead of transmitting raw data, the transmitter sends a (linear) combination of the data, known as coded data. This redundancy in the coded data enables the

Mohammad Ali Maddah-Ali

receiver to recover the raw data even in the presence of errors or missing values. Similarly, coded computing includes three layers [3, 11, 13] (see Figure 1(a)):

The Encoding Layer in which a master node sends a (linear) combination of data, as coded data, to each worker node.

The Computing Layer , in which the worker nodes apply a predefined computation to their assigned coded data and send the results back to the master node.

The Decoding Layer , in which the master node recovers the final results from the computation results over coded data. In this layer, the decoder leverages the coded redundancy in the computation to recover the missing results of the stragglers and detect and correct the adversarial outputs.

The existing coded computing has largely built upon algebraic coding theory, drawing inspiration from the renowned Reed-Solomon code construction in communication [14], with proven straggler and Byzantine resiliency [15]. However, the coding in communication is designed for the exact recovery of the messages, built on a foundation that is inconsistent with the computational requirements of machine learning. Developing a code that preserves its specific construction while composing with computation is extremely challenging, leading to significant restrictions. Firstly, current methods are mainly restricted to specific computation functions, such as polynomials and matrix multiplication [3, 11, 13, 16, 17]. Secondly, rooted in algebraic error correction codes, existing approaches are tailored for finite field computations, leading to numerical instability when dealing with real-valued data [18, 19]. Furthermore, these methods are unsuitable for approximate, fixedpoint, or floating-point computing, where exact computation is neither possible nor necessary, such as in machine learning inference or training tasks. Finally, these schemes typically have a recovery threshold, which is the minimum number of samples required to recover results from coded outputs of worker nodes [3, 13]. If the number of workers falls below this threshold, the recovery process fails entirely.

Several works have attempted to mitigate the aforementioned issues and transform the coded computing scheme into a more robust and adaptable one, applicable to a wide range of computation functions. These efforts include approximating non-polynomial functions with polynomial ones [5, 20], refining the coding mechanism to enhance stability [21-25], and leveraging approximation computing techniques to reduce the recovery threshold and increase recovery flexibility [26-29]. However, these attempts fail to bridge the existing gap between coded computing and general distributed computing systems. The root cause of these issues lies in the fact that they are grounded in coding theory, based on a foundation that is not compatible with the requirements of large-scale machine learning. Therefore, this paper aims to address the following objective:

Objective: The main objective of this paper is to develop a new foundation for coded computing, not solely based on coding theory , but also grounded in learning theory , that seamlessly integrates with machine learning applications, offering a more natural and effective solution for general computing .

In this paper, we establish a learning-theoretic foundation for coded computing, applicable to general computations. We adopt an end-to-end system perspective, that integrates an end-to-end loss function, to find the optimum encoding and decoding functions, focusing on straggler resiliency. We show that the loss function is upper-bounded by the sum of two terms: one characterizing the generalization error of the decoder function and the other capturing the training error of the encoder function. Regularizing the decoder layer, we derive the optimal decoder in the Reproducing Kernel Hilbert space (RKHS) of second-order Sobolev functions. This provides an explicit solution for the optimum decoder function and allows us to characterize the resulting loss of the decoding layer. The decoder loss appears as a regularizing term in optimizing the encoding function and represents the norm in another RKHS. Thus, the optimum solution for the encoding function can be derived, too. We address two noise-free and noisy computation settings, for which we derive the optimal encoder and decoder and corresponding convergence rate. We prove that the proposed framework exhibits a faster convergence rate compared to the state-of-the-art and the numerical evaluations support the theoretical derivations (see Figure 1(b)).

R

3

Encoder

Decoder

3

5

5

2

2

LeNet5architecture,(S,K)=(10,30)

0

-1

-]601

2

-3

4

BACC:slope=-2

LeTCC:slope=-3

4.0

4.5

5.0

5.5

6.0

6.5

log(N)

Contributions: The main contributions of this paper are:

We develop a new foundation for coded computing by integrating it with learning theory, rather than relying solely on coding theory. We define the loss function as the mean square error of the computation estimation, averaged over all possible sets of at most S stragglers (Section 3.1). To be able to find the best encoding and decoding functions, we bound the loss function with the summation of two terms, one characterizing the generalization error of the decoder function and the other capturing the training error of the encoder function (Section 3).

Assuming that the encoder and decoder functions reside in the Hilbert space of secondorder Sobolev functions, we use the theory of RKHSs to find the optimum encoding and decoding functions and characterize the convergence rate for the expected loss in both noise-free and noisy computation regimes (Section 4).

We have extensively evaluated the proposed scheme across different data points and computing functions including state-of-the-art deep neural networks and demonstrated that our proposed framework considerably outperforms the state-of-the-art in terms of recovery accuracy (Section 5).

2 Preliminaries and Problem Definition

2.1 Notations

Throughout this paper, uppercase and lowercase bold letters denote matrices and vectors, respectively. Coded vectors and matrices are indicated by a ∼ sign, as in ˜ x , ˜ A . The set { 1 , 2 , . . . , n } is denoted as [ n ] and symbol | S | denotes the cardinality of the set S . Finally, we represent first, second and k -th order derivative of function f as f ′ , f ′′ , and f ( k ) , respectively.

2.2 Problem Setting

Consider a master node and a set of N workers. The master node is tasked with computing { f ( x k ) } K k =1 using a cluster of N worker nodes, given a set of K data points { x k } K k =1 , x k ∈ R d . Here, f : R d → R m represents an arbitrary function, which could be a simple one-dimensional function or a complex deep neural network, and K,d,m are integers. A naive approach would be to assign the computation of f ( x k ) to one worker node for k ∈ [ K ] . However, some worker nodes may act as stragglers, failing to complete their tasks within the required deadline. To mitigate this issue, the master node employs coding and sends N coded data points to each worker node using an encoder function. Each coded data point is a combination of raw data points. Subsequently, each

Worker Nodes

x1

f(×1)

f(-)

f(x2)

X2

f(-)

f(x1)

X1

f(-)

f(×2)

X2

Encoder

Decoder

Xn

f(xn)

udec(Qk）=f(xk)

f(-)

f(xK)

XK

f(-)

f(×N)

XN

f(-)

worker applies the function f ( · ) to the received coded data and sends the result, coded results, back to the master node. The master node's goal is to approximately recover ˆ f ( x k ) ≈ f ( x k ) using a decoder function, even if some worker nodes appear to be stragglers. The redundancy in the coded data and corresponding coded results enables the master node to recover the desirable results, { f ( x k ) } K k =1 .

3 Proposed Framework: LeTCC

Here, we propose a novel straggler-resistant Learning-Theoretic Coded Computing ( LeTCC ) framework for general distributed computing. As depicted in Figure 2, our framework comprises two encoding and decoding layers, with a computing layer sandwiched between them. The framework operates according to the following steps:

Encoding Layer: The master node fits an encoder regression function u enc : R → R d at points { ( α k , x k ) } K k =1 for fixed, distinct, and ordered values α 1 < α 2 < · · · < α K ∈ R . Then, it computes the encoder function u enc ( · ) on another set of fixed, distinct, and ordered values { β n } N n =1 where β 1 < β 2 < · · · < β N ∈ R , with k ∈ [ K ] and n ∈ [ N ] . Subsequently, the master node sends the coded data points ˜ x n = u enc ( β n ) ∈ R d to worker n for n ∈ [ N ] . Note that each coded data point ˜ x n is a combination of all initial points { x k } K k =1 .

Computing Layer: Each worker node n ∈ [ N ] computes f (˜ x n ) = f ( u enc ( β n )) on its assigned input and sends the result back to the master node.

Decoding Layer: The master node receives the results f (˜ x v ) v ∈F from the non-straggler worker nodes in the set F . Next, it fits a decoder regression function u dec : R → R m at points ( β v , f (˜ x v )) v ∈F = ( β v , f ( u enc ( β v ))) v ∈F . Finally, using the function u dec ( · ) , the master node computes ˆ f ( x k ) := u dec ( α k ) as an approximation of f ( x k ) for k ∈ [ K ] . Recall that u dec ( α k ) ≈ f ( u enc ( α k )) ≈ f ( x k ) .

As mentioned above, the master node selects and fixes the regression points, { α k } K k =1 and { β n } N n =1 , which remain constant throughout the entire process. The encoder and decoder functions are the only components subject to optimization.

Note that the computational efficiency of the encoding and decoding layers is crucial. This includes the fitting process of the encoder and decoder regression functions, as well as the computation of these regression functions at points { β v } v ∈F and { α k } K k =1 . If the master node's computation time is not substantially decreased compared to computing { f ( x k ) } K k =1 by itself, then adopting this framework would not provide any benefits for the master node.

3.1 Objective

We view the whole scheme as a unified predictive framework that provides an approximate estimation of the values { f ( x k ) } K k =1 . We denote the estimator function of the LeTCC scheme as ˆ f α , β [ u enc , u dec , F ]( · ) , where α := [ α 1 , . . . , α K ] T , β := [ β 1 , . . . , β N ] T , and F := { i 1 , . . . , i |F| } represents the set of non-straggler worker nodes.

Let us define a random variable F S,N distributed over the set of all subsets of N workers with maximum S stragglers, {F : F ⊆ [ N ] , |F| ⩾ N -S } . Also, suppose each worker node n ∈ [ N ] computes the function f n ( x ) = f ( x ) + ϵ n , where ϵ n , n ∈ [ N ] are independent zero-mean noise vectors with covariance σ 2 I .

This enables us to define the following loss function, which evaluates the framework's performance:

where ˆ f ( x ) := ˆ f α , β [ u enc , u dec , F ]( x ) to simplify the notation, ∥·∥ 2 represents the ℓ 2 -norm, and ϵ = [ ϵ 1 , . . . , ϵ N ] T . Our objective is to find u enc ( . ) and u dec ( . ) that minimize the objective function (1), which is very challenging, given that ˆ f ( . ) is a composition of u enc ( . ) and u dec ( . ) and the computation in the middle. Here, we take an important step to decompose these two, to gain a deeper understanding of interactions. Adding and subtracting f ( u enc ( α k )) and utilizing inequality of arithmetic and geometric means (AM-GM), one can obtain an upper bound for (1):

The right-hand side of (2) comprises two terms, which uncover an interesting interplay between the encoder and decoder regression functions. Let us elaborate on what each term corresponds to.

L dec ( ˆ f ) -The expected generalization error of the decoder regression: Recall that the master node fits a decoder regression function, u dec ( · ) , at a set of points denoted as { ( β v , f ( u enc ( β v ))) } v ∈F . L dec represents the ℓ 2 -norm of the decoder regression function's error on a distinct set of points { α k } K k =1 , which are different from its training data { β v } v ∈F . Consequently, this term provides an unbiased estimate of the decoder's generalization error. Given that the decoder regression function develops to estimate f ( u enc ( · )) , the generalization error of the decoder regression is inherently tied to the properties of f ( u enc ( · )) . This, in turn, is influenced by characteristics of both the f ( · ) and u enc ( · ) functions, making the L dec ( ˆ f ) a complex interplay of these two functions.

L enc ( ˆ f ) -A proxy to the training error of the encoder regression: Remember that the encoder regression is fitted at points { ( α k , x k ) } K k =1 . Consequently, the training error is calculated as 1 K ∑ K k =1 ∥ u enc ( α k ) -x k ∥ 2 2 . Therefore, L enc represents the encoder training error magnified by the effect of computing function f ( · ) . Specifically, if f ( · ) is q -Lipschitz, then L enc ( ˆ f ) can be upper bounded by:

4 Main Results

In this section, we examine the proposed framework from a theoretical standpoint. We provide a comprehensive explanation of the design process for the decoder and encoder functions and subsequently analyze the convergence rate. For simplicity, we present the results for a one-dimensional

function f : R → R . These results are generalizable to the case where f : R → R m , as discussed in Appendix F.

Suppose the regression points, { α k } K k =1 , { β n } N n =1 , are confined to the interval Ω := ( -1 , 1) and u enc , u dec ∈ ˜ H 2 (Ω; R ) , where ˜ H 2 (Ω; R ) is the reproducing kernel Hilbert space (RKHS) of secondorder Sobolev functions on the interval Ω induced with the norm ∥ f ∥ 2 ˜ H 2 (Ω; R ) := ∫ Ω ( f ′′ ( t )) 2 dt + f ( -1) 2 + f ′ ( -1) 2 which is an equivalent norm on Sobolev space introduced by [30] (see (28) in Appendix A.1). The definition and properties of Sobolev spaces, along with their reproducing kernels and norms, are reviewed in Appendix A.1.

Decoder Design: Since L dec ( ˆ f ) in the decomposition (2) characterizes the generalization error of the decoder function, we propose a regularized objective function for the decoder:

The first term in (4) corresponds to the mean squared error, while the second term characterizes the smoothness of the decoder function on the interval Ω . Equation (4) represents a Kernel Ridge Regression problem (KRR). It can be shown that the solution of (4) has the following form [31, 32]:

where d 0 , d 1 ∈ R , ϕ 0 ( · , · ) is the kernel function of H 2 0 (Ω; R ) (see Definition 2 and (44) in Appendix A.1), and c = [ c 1 , . . . , c |F| ] T ∈ R |F| . Substituting (5) into the main objective (4), the coefficient vectors c and d := [ d 0 , d 1 ] T can be efficiently computed by optimizing a quadratic equation [33]. This solution is known as the second-order smoothing spline function. The theoretical properties of smoothing splines are reviewed in Appendix A.2.

Let us define the following variables, which represent the maximum and minimum distances between consecutive data points in the decoder layer, { β n } N n =1 :

with β 0 := -1 and β N +1 := 1 . The following theorems provide crucial insights for designing the encoder function as well as deriving the convergence rates.

Theorem 1 (Upper bound for noiseless computation, σ 0 = 0 ) . Consider the LeTCC framework with N worker nodes and at most S stragglers with λ d ⩽ N -4 . Assume { α k } K k =1 are arbitrary and distinct points in Ω = ( -1 , 1) and there is constant B such that ∆ max ∆ min ⩽ B . If f ( · ) is a q -Lipschitz continuous function, then:

where C 1 is a constant.

The proof of Theorem 1 and the detailed expression for C 1 can be found in Appendix B.1.

Theorem 2 (Upper bound for noisy computation) . Consider the LeTCC framework with N worker nodes and at most S stragglers and 1 ( N -S ) 4 ⩽ λ d ⩽ λ 0 for constant λ 0 ∈ R . Assume each worker node computes f n ( x ) = f ( x ) + ϵ n with E [ ϵ n ] = 0 and E [ ϵ 2 n ] ⩽ σ 2 0 . Assume { α k } K k =1 are arbitrary and distinct points in Ω = ( -1 , 1) and suppose there is constant B such that ∆ max ∆ min ⩽ B . Assume f ( · ) is a q -Lipschitz continuous function. Then,

if

where C 2 is a constant, C ( λ 0 ) = O ( λ 1 2 0 ) is an increasing function of λ 0 , and p 4 ( S ) is a degree-4 polynomial in S with positive constant coefficients.

The proof and expressions for C 2 and C ( λ 0 ) are provided in Appendix B.2.

Encoder Design: The upper bounds established in Theorems 1, 2 hold for all u enc ∈ ˜ H 2 (Ω; R ) . However, they do not directly lead to a design for u enc ( · ) . To address this, we present the following theorem which bounds the ∥ f ◦ u enc ∥ 2 L 2 (Ω; R ) , enabling us to construct u enc ( · ) without compromising the convergence rate.

Theorem 3. Consider a LeTCC scheme. Assume computing function f ( · ) is q -Lipschitz continuous and ∥ f ′′ ∥ L ∞ (Ω; R ) ⩽ ν . Then:

for some monotonically increasing function ψ : R + → R + , where λ e is depending on ( N,S,σ 0 , q, ν ) .

The proof can be found in Appendix B.3. By applying the representer theorem [34], we can deduce that the optimal encoder u enc ( · ) , which minimizes the right-hand side of (10) takes the form u enc ( · ) = ∑ K k =1 z k ϕ ( α k , · ) , where z ∈ R K , and ϕ is the kernel function of ˜ H 2 (Ω; R ) , as discussed in Appendix A.2 and in (46). However, due to the non-linearity of g ( · ) , calculating the values of the coefficients z is challenging. Nevertheless, we demonstrate that the coefficients can be efficiently derived under certain mild assumptions.

Proposition 1. In the noiseless case, there exists M ∈ R that depends only on { α k } K k =1 and { x k } K k =1 , such that:

See Appendix B.4 for the proof. Proposition 1 states that, under mild assumptions there exists a smoothing spline that minimizes the upper bound given in (12) without changing in convergence rate.

Convergence Rate: Using Theorems 1, 2, and 3, we can derive the convergence rate of the proposed scheme as stated in the following theorem.

Theorem 4 (Convergence rate) . For LeTCC scheme with N worker nodes and a maximum of S stragglers, R ( ˆ f ) ⩽ O ( S 8 5 N -3 5 ) for the noisy computation, and R ( ˆ f ) ⩽ O ( S 3 N -3 ) for the noiseless setting.

Refer to Appendix B.5 for the proof. Notably, the convergence rate yields from Theorem 4 surpasses the state-of-the-art Berrut coded computing approach upper bound [29] (Figure 1), both with respect to S and N (see Appendix C.1 for detailed comparison).

5 Experimental Results

In this section, we extensively evaluate the proposed scheme across various scenarios. Our assessments involve examining multiple deep neural networks as computing functions and exploring the impact of different numbers of stragglers on the scheme's efficiency. The experiments are run using PyTorch [35] in a single GPU machine. We evaluate the performance of the LeTCC scheme in three different model architectures:

Shallow model : We choose LeNet5 [36] architecture as a known shallow network with approximately 6 × 10 4 parameters, trained on the MNIST [37].

Deep model with low-dimensional output : In this scenario, we evaluate the proposed scheme when the function is a deep neural network trained on color images in CIFAR-10 [38] dataset. We use the recently introduced RepVGG [39] network with around 26 million parameters which was trained on CIFAR-10 1 .

Deep model with high-dimensional output : Finally, we demonstrate the performance of the LeTCC scheme in a scenario where the input and output of the computing function are high-dimensional, and the function is a relatively large neural network. We consider the Vision Transformer (ViT) [40] as one of the state-of-the-art base neural networks in computer vision for our prediction model, with more than 80 million parameters (in the base version). The network was trained and fine-tuned on the ImageNet-1K dataset [41] 2 .

We use the output of the last softmax layer of each model as the output.

Hyper-parameters: The entire encoding and decoding process is the same for different functions, as we adhere to a non-parametric approach. The sole hyper-parameters involved are the two smoothing parameters ( λ enc , λ dec) which are determined using cross-validation and greed search over different values of the smoothing parameters.

Baseline: We compare LeTCC with the Berrut approximate coded computing ( BACC ) introduced by [29] as the state-of-the-art coded computing scheme for general computing. The BACC framework is used in [29] for training neural networks and in [42] for inference. Although Berrut coded computing [29] is the only existing coded computing scheme for general functions, we include a comparison of the proposed framework with the Lagrange coded computing scheme [3] for polynomial computation in Appendix D.

Interpolation Points: We choose Chebyshev points of the first and second kind, { α i } K k =1 = cos ( (2 k -1) π 2 K ) and { β n } N n =1 = cos ( ( n -1) π N -1 ) , for fair comparison with [29].

Evaluation Metrics: We employ two evaluation metrics to assess the performance of the proposed framework: Relative Accuracy (RelAcc) and Root Mean Squared Error (RMSE). RelAcc is defined as the ratio of the base model's prediction accuracy to the accuracy of the estimated model on the initial data points. RMSE, on the other hand, is our main loss defined in (1) which measures the empirical average of the root mean square difference over multiple batches of and non-straggler set F , providing an unbiased estimation of expected mean square error, E x ∼X , F [ 1 K ∑ K k =1 ∥ f ( x k ) -ˆ f ( x k ) ∥ 2 ] , for data distribution X .

ViT architecture,(N,K)=(20,14)

RepVGG architecture,(N,K)=(60,45)

LeNet5architecture,(N,K)=(100,60)

1.3

4.5

BACC

BACC

BACC

1.2

LeTCC

3.5

LeTCC

4.0

LeTCC

1.1

3.5

[2]3

10

[ ]3

3.0

[]3

3.0

0.9

2.5

2.5

0.8

2.0

2.0

0.7

1.5

0.6

1.5

Number of Stragglers

2

3

4

Number of Stragglers

6

8

10

12

14

5

10

Number of Stragglers

15

20

25

30

35

ViT architecture,(N,K)=(20,14)

RepVGG architecture,(N,K)=(60,45)

LeNet5 architecture,(N,K)=(100,60)

1.1

1.00

1.00

1.0

S60

0.9

0.90

S60

0.85

0.90

RelAcc

0.85

0.7

SL0

0.80

0.6

0.70

0.5

BACC

0.65

BACC

0.75 -

BACC

LeTCC

0.60

LeTCC

0.70

LeTCC

0.4

2

3

4

8

10

12

14

5

10

15

20

25

OE

35

Number of Stragglers

Number of Stragglers

Number of Stragglers

Performance Evaluation: Table 1 presents both RMSE and RelAcc metrics side by side. The results demonstrate that LeTCC outperforms BACC across various architectures and configurations, with an average improvement of 15%, 17%, and 9% in RMSE for LeNet, RepVGG, and ViT architectures, respectively, and a 2%, 5%, and 4% enhancement in RelAcc.

In a subsequent analysis, we evaluate the performance of LeTCC in comparison to BACC across a variety of straggler scenarios. For each number of stragglers, S , we randomly select S workers to act as stragglers. Both schemes are then run with the same input data points and straggler configurations, and the process is repeated 20 times. We record the average values of the RelAcc and RMSE metrics, along with their 95% confidence intervals. Figures 3 and 4 illustrate the performance of both schemes across three model architectures. As shown in both figures, the proposed scheme consistently outperforms BACC for nearly all straggler values. In Figure 3, where N K is relatively small-indicating a system design without excessive redundancy, which is more practical-the proposed scheme demonstrates even greater improvements in both metrics.

Computational Complexity: The calculation and inference of smoothing spline coefficients can be performed linearly in the number of regression points by leveraging B-spline basis functions [43-45]. Consequently, the encoding and decoding processes in LeTCC , which involve evaluating new points and calculating the fitted coefficients, have computational complexities of O ( K · d ) and O (( N -S ) · m ) , respectively, where d is the input dimension and m is the output dimension of the computing function f ( · ) . This complexity is comparable to that of BACC , which has complexities of O ( K ) and O ( N -S ) for its encoding and decoding layers [29]. Table 2 in Appendix C.2 presents a comparison of the total end-to-end processing time statistics for the LeTCC and BACC schemes.

Sensitivity Analysis: We additionally investigate the sensitivity of the proposed schemes performance to the value of the smoothing parameter, as well as the sensitivity of the optimal smoothing parameter to the number of stragglers (or workers). The results are presented in Appendix E. As shown in Table 3 and Figure 6, the optimal smoothing parameters and the scheme's performance exhibit low sensitivity to the number of stragglers (or worker nodes) and to the smoothing parameters, respectively.

Coded Points: We also compare the coded points { ˜ x n } N n =1 sent to the workers in LeTCC and BACC schemes. The results, shown in Figure 7, demonstrate that BACC coded samples exhibit highfrequency noise which causes the scheme to approximate the original prediction worse than LeTCC .

ViT architecture,(N,K)=(20,8)

RepVGG architecture, (N,K)=(60,20)

LeNet5 architecture,(N,K) =(100,20)

1.8

BACC

BACC

BACC

LeTCC

LeTCC

LeTCC

1.6

1.4

[7]3

121

[7 ]3

[1]3

1.0

0.8

2

0.6

0.4

2

Number of Stragglers

4

6

10

12

10

Number of Stragglers

15

25

30

35

10

20

Number of Stragglers

30

40

60

70

ViT architecture,(N,K)=(20,8)

RepVGG architecture,(N,K)=(60,20)

LeNet5 architecture, (N,K) = (100,20)

1.1

L.1

1.0

1.0

1.0

0.9

60

60

RelAcc

0.8

RelAcc

0.8

0.7

0.6

0.5

0.6

0.7

BACC

BACC

BACC

0.4

LeTCC

0.5

LeTCC

LeTCC

90

4

10

ZT

10

15

20

30

35

10

20

30

40

60

70

Number of Stragglers

Number of Stragglers

Number of Stragglers

6 Related Work

Coded computing was initially introduced to tackle the challenges of distributed computation, particularly the existence of stragglers or slow workers, and also faulty or adversarial nodes. Traditional approaches to deal with stragglers primarily rely on repetition [1, 46-50], where each task is assigned to multiple workers either proactively or reactively. Recently, coded computing approaches have reduced the overhead of repetition by leveraging coding theory and embedding redundancy in the worker's input data [3, 13, 26, 29, 51-55]. This technique, which mainly relies on theory of coding, has been developed for specific types of structured computations, such as polynomial computation and matrix multiplication [3, 7, 12, 13, 17, 52, 56-58]. Recently, there have been attempts to generalize coded computing for general computations [4, 5, 29, 59]. Towards extending the application of coding computing to machine learning computation, Kosaian et al. [59] suggest training a neural network to predict coded outputs from coded data points. However, the scheme of Kosaian et al. [59] requires a complex training process and tolerates only one straggler. In another work, JahaniNezhad and Maddah-Ali [29] proposes BACC , a model-agnostic and numerically stable framework for general computations. They successfully employed BACC to train neural networks on a cluster of workers, while tolerating a larger number of stragglers. Building on the BACC framework, Soleymani et al. [42] introduced ApproxIFER scheme, as a straggler resistance and Byzantine-robust prediction serving system. However, the scheme of BACC uses a reasonable rational interpolation (Berrut interpolation [60]), off the shelf, for encoding and decoding, without considering any end-to-end cost function to optimize. In contrast, we theoretically formalize a new foundation of coded computing grounded in learning theory, which can be naturally used for machine learning applications.

7 Conclusions and Future Work

In this paper, we developed a new foundation for coded computing based on learning theory, contrasting with existing works that rely on coding theory and use metrics like minimum distance and recovery threshold for design. This shift in foundations removes barriers to using coded computing for machine learning applications, allows us to design optimal encoding and decoding functions, and achieves convergence rates that outperform the state of the art. Moreover, the experimental evaluations validate the theoretical guarantees. While this work focuses on straggler mitigation, future work will extend our proposed scheme to achieve Byzantine robustness and privacy, offering promising avenues for further research.

8 Acknowledgments

This material is based upon work supported by the National Science Foundation under Grant CIF2348638. Behrooz Tahmasebi is supported by NSF Award CCF-2112665 (TILOS AI Institute) and NSF Award 2134108.

References

Jeffrey Dean and Luiz André Barroso. The tail at scale. Communications of the ACM , 56(2): 74-80, 2013.

Vipul Gupta, Shusen Wang, Thomas Courtade, and Kannan Ramchandran. Oversketch: Approximate matrix multiplication for the cloud. In 2018 IEEE International Conference on Big Data (Big Data) , pages 298-304. IEEE, 2018.

Qian Yu, Songze Li, Netanel Raviv, Seyed Mohammadreza Mousavi Kalan, Mahdi Soltanolkotabi, and Salman A Avestimehr. Lagrange coded computing: Optimal design for resiliency, security, and privacy. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1215-1225. PMLR, 2019.

Jinhyun So, Basak Guler, and Amir Salman Avestimehr. A scalable approach for privacypreserving collaborative machine learning. In Advances in Neural Information Processing Systems , volume 33, pages 8054-8066, 2020.

Jinhyun So, Ba¸ sak Güler, and Amir Salman Avestimehr. CodedPrivateML: A fast and privacypreserving framework for distributed machine learning. IEEE Journal on Selected Areas in Information Theory , 2(1):441-451, 2021.

Zhuqing Jia and Syed Ali Jafar. On the capacity of secure distributed batch matrix multiplication. IEEE Transactions on Information Theory , 67(11):7420-7437, 2021.

Malihe Aliasgari, Osvaldo Simeone, and Jörg Kliewer. Private and secure distributed matrix multiplication with flexible communication load. IEEE Transactions on Information Forensics and Security , 2020.

Minchul Kim and Jungwoo Lee. Private secure coded computation. In 2019 IEEE International Symposium on Information Theory (ISIT) , pages 1097-1101. IEEE, 2019.

Wei-Ting Chang and Ravi Tandon. On the upload versus download cost for secure and private matrix multiplication. In 2019 IEEE Information Theory Workshop (ITW) , pages 1-5. IEEE, 2019.

Wei-Ting Chang and Ravi Tandon. On the capacity of secure distributed matrix multiplication. In 2018 IEEE Global Communications Conference (GLOBECOM) , pages 1-6, 2018.

Qian Yu, Mohammad Ali Maddah-Ali, and A Salman Avestimehr. Straggler mitigation in distributed matrix multiplication: Fundamental limits and optimal coding. IEEE Transactions on Information Theory , 66(3):1920-1933, 2020.

Rafael G.L. DOliveira, Salim El Rouayheb, and David Karpuk. GASP codes for secure distributed matrix multiplication. IEEE Transactions on Information Theory , 66(7):4038-4050, 2020.

Qian Yu, Mohammad Maddah-Ali, and Salman Avestimehr. Polynomial codes: an optimal design for high-dimensional coded matrix multiplication. Advances in Neural Information Processing Systems , 30, 2017.

Irving S Reed and Gustave Solomon. Polynomial codes over certain finite fields. Journal of the society for industrial and applied mathematics , 8(2):300-304, 1960.

Venkatesan Guruswami, Atri Rudra, and Madhu Sudan. Essential Coding Theory . Draft is Available, 2022.

K. Lee, M. Lam, R. Pedarsani, D. Papailiopoulos, and K. Ramchandran. Speeding up distributed machine learning using codes. IEEE Transactions on Information Theory , 64(3):15141529, 2018.

K. Lee, C. Suh, and K. Ramchandran. High-dimensional coded matrix multiplication. In Proceedings of IEEE International Symposium on Information Theory (ISIT) , pages 2418-2422, 2017.

Walter Gautschi and Gabriele Inglese. Lower bounds for the condition number of vandermonde matrices. Numerische Mathematik , 52(3):241-250, 1987.

Gene H. Golub and Charles F. Van Loan. Matrix computations . JHU press, 2013.

Jinhyun So, Basak Guler, and Salman Avestimehr. A scalable approach for privacy-preserving collaborative machine learning. Advances in Neural Information Processing Systems , 33:80548066, 2020.

Adarsh M. Subramaniam, Anoosheh Heidarzadeh, and Krishna R. Narayanan. Random Khatri-Rao-product codes for numerically-stable distributed matrix multiplication. CoRR , abs/1907.05965, 2019.

Aditya Ramamoorthy and Li Tang. Numerically stable coded matrix computations via circulant and rotation matrix embeddings. IEEE Transactions on Information Theory , 68(4):2684-2703, 2022.

Anindya Bijoy Das, Aditya Ramamoorthy, and Namrata Vaswani. Efficient and robust distributed matrix computations via convolutional coding. IEEE Transactions on Information Theory , 67(9):6266-6282, 2021.

Mahdi Soleymani, Hessam Mahdavifar, and Amir Salman Avestimehr. Analog Lagrange coded computing. IEEE Journal on Selected Areas in Information Theory , 2(1):283-295, 2021.

Mohammad Fahim and Viveck R Cadambe. Numerically stable polynomially coded computing. IEEE Transactions on Information Theory , 67(5):2758-2785, 2021.

Tayyebeh Jahani-Nezhad and Mohammad Ali Maddah-Ali. CodedSketch: A coding scheme for distributed computation of approximated matrix multiplication. IEEE Transactions on Information Theory , 67(6):4185-4196, 2021.

Vipul Gupta, Shusen Wang, Thomas Courtade, and Kannan Ramchandran. OverSketch: Approximate matrix multiplication for the cloud. In 2018 IEEE International Conference on Big Data (Big Data) , pages 298-304, 2018.

Vipul Gupta, Swanand Kadhe, Thomas Courtade, Michael W. Mahoney, and Kannan Ramchandran. Oversketched Newton: Fast convex optimization for serverless systems. In 2020 IEEE International Conference on Big Data (Big Data) , pages 288-297, 2020.

Tayyebeh Jahani-Nezhad and Mohammad Ali Maddah-Ali. Berrut approximated coded computing: Straggler resistance beyond polynomial computing. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(1):111-122, 2022.

George Kimeldorf and Grace Wahba. Some results on tchebycheffian spline functions. Journal of mathematical analysis and applications , 33(1):82-95, 1971.

Jean Duchon. Splines minimizing rotation-invariant semi-norms in sobolev spaces. In Constructive Theory of Functions of Several Variables: Proceedings of a Conference Held at Oberwolfach April 25-May 1, 1976 , pages 85-100. Springer, 1977.

Grace Wahba. Spline models for observational data . SIAM, 1990.

Grace Wahba. Smoothing noisy data with spline functions. Numerische mathematik , 24(5): 383-393, 1975.

Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. A generalized representer theorem. In International conference on computational learning theory , pages 416-426. Springer, 2001.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Yann LeCun, Corinna Cortes, Chris Burges, et al. MNIST handwritten digit database, 2010.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13733-13742, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Mahdi Soleymani, Ramy E Ali, Hessam Mahdavifar, and A Salman Avestimehr. ApproxIFER: A model-agnostic approach to resilient and robust prediction serving systems. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8342-8350, 2022.

Paul HC Eilers and Brian D Marx. Flexible smoothing with b-splines and penalties. Statistical science , 11(2):89-121, 1996.

Carl De Boor. Calculation of the smoothing spline with weighted roughness measure. Mathematical Models and Methods in Applied Sciences , 11(01):33-41, 2001.

Chiu Li Hu and Larry L Schumaker. Complete spline smoothing. Numerische Mathematik , 49: 1-10, 1986.

Matei Zaharia, Andy Konwinski, Anthony D Joseph, Randy H Katz, and Ion Stoica. Improving mapreduce performance in heterogeneous environments. In Osdi , volume 8, page 7, 2008.

Lalith Suresh, Marco Canini, Stefan Schmid, and Anja Feldmann. C3: Cutting tail latency in cloud data stores via adaptive replica selection. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15) , pages 513-527, 2015.

Nihar B Shah, Kangwook Lee, and Kannan Ramchandran. When do redundant requests reduce latency? IEEE Transactions on Communications , 64(2):715-722, 2015.

Kristen Gardner, Samuel Zbarsky, Sherwin Doroudi, Mor Harchol-Balter, and Esa Hyytia. Reducing latency via redundant requests: Exact analysis. ACM SIGMETRICS Performance Evaluation Review , 43(1):347-360, 2015.

Manmohan Chaubey and Erik Saule. Replicated data placement for uncertain scheduling. In 2015 IEEE International Parallel and Distributed Processing Symposium Workshop , pages 464-472. IEEE, 2015.

Songze Li, Seyed Mohammadreza Mousavi Kalan, Amir Salman Avestimehr, and Mahdi Soltanolkotabi. Near-optimal straggler mitigation for distributed gradient methods. In 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) , pages 857-866. IEEE, 2018.

Anindya B Das, Aditya Ramamoorthy, and Namrata Vaswani. Random convolutional coding for robust and straggler resilient distributed matrix computation. arXiv preprint arXiv:1907.08064 , 2019.

Aditya Ramamoorthy, Anindya Bijoy Das, and Li Tang. Straggler-resistant distributed matrix computation via coding theory: Removing a bottleneck in large-scale data processing. IEEE Signal Processing Magazine , 37(3):136-145, 2020.

Can Karakus, Yifan Sun, Suhas Diggavi, and Wotao Yin. Straggler mitigation in distributed optimization through data encoding. Advances in Neural Information Processing Systems , 30, 2017.

R. Tandon, Q. Lei, A. G. Dimakis, and N. Karampatziakis. Gradient coding: Avoiding stragglers in distributed learning. In Proceedings of the 34th International Conference on Machine Learning (ICML) , pages 3368-3376, Sydney, Australia, August 2017.

Aditya Ramamoorthy and Li Tang. Numerically stable coded matrix computations via circulant and rotation matrix embeddings. IEEE Transactions on Information Theory , 68(4):2684-2703, 2021.

Qian Yu, Mohammad Ali Maddah-Ali, and Amir Salman Avestimehr. Straggler mitigation in distributed matrix multiplication: Fundamental limits and optimal coding. IEEE Transactions on Information Theory , 66(3):1920-1933, 2020.

Mohammad Fahim and Viveck R. Cadambe. Numerically stable polynomially coded computing. IEEE Transactions on Information Theory , 67(5):2758-2785, 2021.

Jack Kosaian, KV Rashmi, and Shivaram Venkataraman. Parity models: A general framework for coding-based resilience in ml inference. arXiv preprint arXiv:1905.00863 , 2019.

J-P Berrut. Rational functions for guaranteed and experimentally well-conditioned global interpolation. Computers & Mathematics with Applications , 15(1):1-16, 1988.

Giovanni Leoni. Afirst course in Sobolev spaces , volume 181. American Mathematical Society, 2024.

Robert A Adams and John JF Fournier. Sobolev spaces . Elsevier, 2003.

Alain Berlinet and Christine Thomas-Agnan. Reproducing kernel Hilbert spaces in probability and statistics . Springer Science & Business Media, 2011.

David L Ragozin. Error bounds for derivative estimates based on spline smoothing of exact or noisy data. Journal of approximation theory , 37(4):335-355, 1983.

Florencio I Utreras. Convergence rates for multivariate smoothing spline functions. Journal of approximation theory , 52(1):1-27, 1988.

Heinrich W Guggenheimer, Alan S Edelman, and Charles R Johnson. A simple estimate of the condition number of a linear system. The College Mathematics Journal , 26(1):2-5, 1995.

A Preliminaries

A.1 Sobolev spaces and Sobolev norms

Let Ω be an open interval in R and M be a positive integer. We denote by L p ( Ω; R M ) the class of all measurable functions g : R → R M that satisfy:

where g ( · ) = [ g 1 ( · ) , . . . , g M ( · )] T . The space L p ( Ω; R M ) can be endowed with the following norm, known as the L p norm:

for 1 ⩽ p < ∞ , and

Definition 1 ( Sobolev Space ) . The Sobolev space W m,p ( Ω; R M ) is the space of all functions g ∈ L p ( Ω; R M ) such that all weak derivatives of order i , denoted by g ( i ) , belong to L p ( Ω; R M ) for i ∈ [ m ] . This space is endowed with the norm:

for p = ∞ . Additionally, a function g : Ω → R M is in L p loc ( Ω; R M ) if it lies in L p ( V ; R M ) for all compact subsets V ⊆ Ω .

for 1 ⩽ p < ∞ , and for p = ∞ .

Definition 2. ( Sobolev Space with compact support ): Denoted by W m,p 0 ( Ω; R M ) collection of functions g defined on interval Ω = ( a, b ) such that g ( a ) = 0 , g ′ ( a ) = 0 , . . . , g ( m -1) ( a ) = 0 and ∥ ∥ g ( m ) ∥ ∥ L 2 (Ω; R ) < ∞ . This space can be endowed with the following norm:

Similarly, W m,p loc ( Ω; R M ) is defined as the space of all functions g ∈ L p loc ( Ω; R M ) with all weak derivatives of order i belonging to L p loc ( Ω; R M ) for i ∈ [ m ] . ∥ g ∥ W m,p loc (Ω; R M ) and ∥ g ∥ W m, ∞ loc (Ω; R M ) are defined similar to (16) and (19) respectively, using L p loc (Ω; R ) instead of L p (Ω; R ) .

for 1 ⩽ p < ∞ , and for p = ∞ .

The next theorem provides an upper bound for L p norm of functions in the Sobolev space, which plays a crucial role in the proof of the main theorems of the paper.

Theorem 5 (Theorem 7.34, [61]) . Let Ω ⊆ R be an open interval and let g ∈ W 1 , 1 loc ( Ω; R M ) . Assume 1 ⩽ p, q, r ⩽ ∞ and r ⩾ q . Then:

for every 0 < ℓ < L 1 (Ω) .

Note that L 1 (Ω) in Theorem 5 is the length of the interval Ω .

Corollary 1. Suppose g ∈ W 1 , 1 loc ( Ω; R M ) and Ω ⊆ R be an open interval. If ∥ g ∥ L 2 ( Ω; R M ) ∥ g ′ ∥ L 2 ( Ω; R M ) < L 1 (Ω) , then:

Proof. Substituting p, q = 2 and r = ∞ in Theorem 5 and optimizing over ℓ , one can derive the optimum value of ℓ , denoted by ℓ ∗ as

Since ∥ g ∥ L 2 ( Ω; R M ) ∥ g ′ ∥ L 2 ( Ω; R M ) < L 1 (Ω) , the optimum value is in the valid interval mentioned in Theorem 5.

Corollary 2 (Corollary 7.36, [61]) . Let Ω = ( a, b ) , let 1 ≤ p, q, r ≤ ∞ be such that 1 + 1 /r ≥ 1 /p and r ≥ q and let g ∈ W 1 , 1 loc ( Ω; R M ) with g ′ ∈ L p ( Ω; R M ) . Let x 0 ∈ [ a, b ] be such that | g ( x 0 ) | = min [ a,b ] | g | . Then

where α := 0 if r = q and 1 -1 /p +1 /r = 0 and otherwise

Proof. We prove the above corollary by showing that W 2 , 2 ( Ω; R M ) ⊆ W 1 , 1 ( Ω; R M ) ⊆ W 1 , 1 loc ( Ω; R M ) . Using Cauchy-Schwartz inequality, one can show:

Corollary 3. Theorem 5 and Corollary 2 hold true when f ∈ W 2 , 2 ( Ω; R M ) .

where (a) follows from the bounded length of Ω and f ∈ W 2 , 2 (Ω; R ) . Similarly:

Equations (24) and (25) prove that W 2 , 2 ( Ω; R M ) ⊆ W 1 , 1 ( Ω; R M ) . Now, suppose f ∈ W 1 , 1 ( Ω; R M ) . For every closed subset [ c, d ] ⊂ Ω we have:

where (a) is due to [ c, d ] ⊂ Ω and (b) is because of f ∈ W 1 , 1 ( Ω; R M ) . Therefore, f ∈ W 1 , 1 loc ( Ω; R M ) .

Equivalent norms. There have been various norms defined on Sobolev spaces in the literature that are equivalent to (16) (see [62], [63, Ch. 7], and [32, Sec. 10.2]). Note that two norms ∥·∥ W 1 , ∥·∥ W 2 are equivalent if there exist positive constants η 1 , η 2 such that η 1 · ∥ g ∥ W 2 ⩽ ∥ g ∥ W 1 ⩽ η 2 · ∥ g ∥ W 2 . The equivalent norm in which we are interested is the one introduced in [30]. Let Ω = ( a, b ) ⊂ R . We define W m,p ( Ω; R M ) as the Sobolev space endowed with the following norm:

The following lemma derives the equivalence constants ( η 1 , η 2 ) for the norms ∥·∥ W 2 , 2 (Ω; R ) and ∥·∥ W 2 , 2 (Ω; R ) .

˜ Lemma 1. Let Ω = ( a, b ) be an arbitrary open interval in R . Then for every g ∈ W 2 , 2 (Ω; R ) :

Proof. By expanding g around the point a and utilizing the integral remainder form of Taylor's expansion, for every x ∈ ( a, b ) we have:

Therefore:

where (a) follows from the 2 g ( a ) · ∫ x a g ′ ( t ) dt ⩽ g ( a ) 2 + (∫ x a g ′ ( t ) dt ) 2 , (b) is based on CauchySchwartz inequality, and (c) is due to x ⩽ b . Following the same steps as (32), we have:

Integrating both sides of (33) we have:

where (a) is based on adding the positive term ( b -a ) · g ( a ) 2 to the right-hand side. Based on the (32) and (34) we have:

˜ where (a) follows by integrating both sides of (32) and (b) is due to (33). Using (35) and (34) and the fact that ∫ b a g ′′ ( t ) 2 dt ⩽ ∥ g ∥ 2 W 2 , 2 (Ω; R ) , we have:

where (a) is because of -2 g ( a ) · ∫ x a g ′ ( t ) dt ⩽ g ( a ) 2 + (∫ x a g ′ ( t ) dt ) 2 , (b) follows from CauchySchwartz inequality, and (c) is due to x ⩽ b . Integrating both sides of (37), we have

where (a) follows by a change of variable from x to t and (b) follows by adding the positive term 2 max { 1 , ( b -a ) } 2 · ∫ b a g ′′ ( t ) 2 dt to the right side. Thus, (38) directly results in

Following same steps as (38) and (39) results in

Considering the fact that ∫ b a g ′′ ( t ) 2 dt ⩽ ∥ g ∥ 2 W 2 , 2 (Ω; R ) , we can complete the proof:

Corollary 4. Based on Lemma 1, for Ω = ( -1 , 1) we have

Corollary 5. The result of Lemma 1 remains valid for multi-dimensional cases, where g : R → R M , for some M > 1 .

Corollary 4 directly follows from Lemma 1 by substituting ( a, b ) = ( -1 , 1) .

Corollary 5 directly follows from applying Lemma 1 to each component of the function g ( · ) and using the definition of vector-valued function norm:

Proposition 2. ([61, Section 7.2], [63, Theorem 121],[32]) For any open interval Ω ⊆ R and m,M ∈ N , are Reproducing Kernel Hilbert Spaces (RKHSs).

The full expression of the kernel function of H m (Ω; R ) and ˜ H m (Ω; R ) and other equivalent norms of Sobolev spaces can be found in [63, Section 4]. For H m (Ω; R ) the kernel function is as follows:

where ( · ) + is positive part function.

A.2 Smoothing Splines

Consider the data model y i = f ( t i ) + ϵ i for i = 1 , . . . , n , where t i ∈ Ω = ( a, b ) ⊂ R , E [ ϵ i ] = 0 , and E [ ϵ 2 i ] ⩽ σ 2 0 . Assuming f ∈ ˜ W m, 2 (Ω; R ) , the solution to the following optimization problem is referred to as the smoothing spline:

where y = [ y 1 , . . . , y n ] . Based on Proposition 2, ˜ H m (Ω; R ) := ˜ W m, 2 (Ω; R ) with the norm ∥·∥ ˜ W m, 2 (Ω; R ) is a RKHS for some kernel function ϕ ( · , · ) . Therefore for any v ∈ ˜ W m, 2 (Ω; R ) , we have:

v ( t ) = ⟨ v ( · ) , ϕ ( · , t ) ⟩ ˜ H m (Ω; R ) . (46) It can be shown that ϕ ( t, s ) = R P ( t, s ) + R 0 ( t, s ) where R 0 ( t, s ) is kernel function of H m 0 (Ω; R ) and R P ( t, s ) is a null space of H m 0 (Ω; R ) which is the space of all polynomials with degree less than m .

The solution of (45) has the following form [31, 32]:

where ν j ( · ) = R 0 ( · , t j ) for j ∈ [ n ] , and { ζ i ( · ) } m i =1 are the basis functions of the space of polynomials of degree at most m -1 . Substituting u ∗ into (45) and optimizing over c = [ c 1 , . . . , c n ] T and d = [ d 1 , . . . , d m ] T , we obtain the following result [32]:

where

Equation (48) states that the smoothing spline fitted on the data points y is a linear operator:

for z ∈ R n , where A λ := Q ( Q T Q + λ Γ ) -1 Q T . It can be shown that the u ∗ ( · ) is a natural spline [32, 33]. Thus, if { b i ( · ) } n i =1 is a basis function for an m -th order natural spline (such as truncated power or B-spline basis functions), we have:

where N ij = b i ( t j ) , Φ ij = ∫ Ω b ′′ i ( t ) b ′′ j ( t ) dt for i, j ∈ [ n ] , and ξ := [ ξ 1 , . . . , ξ n ] T .

To characterize the estimation error of the smoothing spline, | f -S λ,n,m ( y ) | , we need to define two variables analogous to those in (6), which quantify the minimum and maximum consecutive distance of the regression points { t i } n i =1 :

where boundary points are defined as ( t 0 , t n +1 ) := ( a, b ) . The following theorem offers an upper bound for the j -th derivative of the smoothing spline estimator error function in the absence of noise ( σ 0 = 0 ).

Theorem 6. ([64, Theorem 4.10]) Consider data model y i = f ( t i ) with { t i } n i =1 belong to Ω = [ a, b ] for i ∈ [ n ] . Let where p d ( · ) is a degree d polynomial with positive weights and D ( m ) is a function of m . Then for each j ∈ { 0 , 1 , . . . , m } and any f ∈ W m, 2 (Ω; R ) , there exist a function H ( m,j ) such that:

Note that ( f -S λ,n,m ( y )) (0) := f -S λ,n,m ( y ) . In the presence of noise, where σ 0 > 0 , we can exploit the linearity of the smoothing spline operator and the mutual independence of the noise terms to conclude that:

where f = [ f ( t 1 ) , . . . , f ( t n )] T . The first term in (56) can be upper bounded using Theorem 6. The following theorem establishes an upper bound for the second term when ∆ max ∆ min is bounded:

Theorem 7. ([65, Section 5]) Consider data model y i = f ( t i )+ ϵ i , where E [ ϵ i ] = 0 and E [ ϵ 2 i ] ⩽ σ 2 0 for i ∈ [ n ] . Assume there exist a constant B > 0 such that ∆ max ∆ min ⩽ B . Then for each j ∈ { 0 , 1 , . . . , m } , there exist a constant λ 0 > 0 and function Q ( m,j, λ 0 ) such that:

for λ ⩽ λ 0 and nλ 1 2 m ⩾ 1 .

Note that based on [65], Q ( m,j, λ 0 ) = w ( m,j ) λ 1 2 m 0 + ˜ w ( m,j ) .

B Proof of Theorems

Recall from (2) that R ( ˆ f ) ⩽ L enc ( ˆ f ) + L dec ( ˆ f ) , where

We begin by deriving a general intermediate bound for L dec and L enc which will be a key component in the proofs of Theorems 2 and 1. The subsequent subsections will then provide the remaining details to complete the proofs of both theorems.

Lemma 2. Let f : R → R be a q -Lipschitz continuous function. Then:

Proof. Using Lipschitz property, we have:

As previously mentioned, L dec represents the expected generalization error of the decoder function. To leverage the results from Theorems 6 and 7 we must establish that the composition of f with the encoder u enc belongs to the Sobolev space W 2 , 2 (Ω; R ) .

Lemma 3. Let f : R → R be a q -Lipschitz continuous function with ∥ f ′′ ∥ L ∞ (Ω; R ) ⩽ ν and Ω ⊂ R be an open interval. If u enc ∈ W 2 , 2 (Ω; R ) then f ◦ u enc ∈ W 2 , 2 (Ω; R ) .

Proof. Let us define f 0 ( t ) := f ( t ) -f (0) . Thus f 0 (0) = 0 and f 0 is q -Lipschitz. Using Lipschitz property

Integrating both sides of (62):

where (a) follows by u enc ∈ W 2 , 2 (Ω; R ) . Given that f 0 is q -Lipschitz, its derivative is bounded in the L ∞ (Ω; R ) -norm, i.e., ∥ f ′ 0 ∥ L ∞ (Ω; R ) ⩽ q . Thus

where (a) follows by chain rule and (b) follows by u enc ∈ W 2 , 2 (Ω; R ) . For the second derivative we have:

where (a) follows from the chain rule, (b) is derived using the Cauchy-Schwartz inequality, (c) is due to ∥ f ′′ 0 ∥ L ∞ (Ω; R ) ⩽ ν , (d) follows from Theorem 5 with r = 4 , p = q = 2 , l = 1 , and (e) is a result of u enc ∈ W 2 , 2 (Ω; R ) . Equations (63), (64), and (63) demonstrate that f 0 ◦ u enc ∈ W 2 , 2 (Ω; R ) . Note that Ω is bounded, and every constant function belongs to W 2 , 2 (Ω; R ) . Thus, we can conclude that f 0 ◦ u enc ( t ) + f (0) = f ◦ u enc ( t ) ∈ W 2 , 2 (Ω; R ) .

Let us define the function h ( t ) := u dec ( t ) -f ( u enc ( t )) . Based on Lemma 3, and given that u dec and f ◦ u enc belong to the Sobolev space W 2 , 2 (Ω; R ) , it follows that h ∈ W 2 , 2 (Ω; R ) . The subsequent lemmas establish upper bounds for ∥ h ∥ L ∞ (Ω; R ) and ∥ h ′ ∥ L ∞ (Ω; R ) , leveraging properties of functions in Sobolev spaces.

Lemma 4. If Ω = ( -a, a ) , then:

Proof. Assume ∃ x 0 ∈ Ω : h ( x 0 ) = 0 . Therefore, | h ( x ) | = ∣ ∣ ∣ ∫ x x 0 h ′ ( x ) dx ∣ ∣ ∣ for x ∈ [ x 0 , a ) . Thus

where (a) and (b) are followed by the triangle and Cauchy-Schwartz inequalities respectively. Integrating the square of both sides over the interval [ x 0 , a ) yields:

where (a) follows by x 0 < a . On the other side, we have the following for every x ∈ ( -a, x 0 ] :

Therefore, we have a similar inequality:

Using (68) and (70) completes the proof:

Thus, if x 0 exists, the proof is complete. In the next step, we prove the existence of x 0 ∈ Ω such that h ( x 0 ) = 0 . Recall that h ( t ) = u dec ( t ) -f ( u enc ( t )) and u dec ( · ) is the solution of (4). Assume there is no such x 0 . Since h ∈ W 2 , 2 (Ω; R ) , then h ( · ) is continuous. Therefore, if there exist t 1 , t 2 ∈ Ω such that h ( t 1 ) < 0 and h ( t 2 ) > 0 , then the intermediate value theorem states that there exists x 0 ∈ ( t 1 , t 2 ) such that h ( x 0 ) = 0 . Thus, h ( t ) > 0 or h ( t ) < 0 for all t ∈ Ω . Without loss of generality, assume the first case where h ( t ) > 0 for all t ∈ Ω . It means that u dec ( t ) > f ( u enc ( t )) for all t ∈ Ω . Let us define where (a) follows from u dec ( β ∗ ) > 0 and ¯ u dec ( β v ) > f ( u enc ( β v )) for all v ∈ F . This leads to a contradiction since it implies that u dec is not the solution of the (4). Therefore, our initial assumption must be wrong. Thus, there exists x 0 ∈ Ω such that h ( x 0 ) = 0 .

Lemma 5. Let Ω = ( -1 , 1) . For h ( t ) = u dec ( t ) -f ( u enc ( t )) we have:

Proof. Using Lemma 4 one can conclude

Since h ∈ W 2 , 2 (Ω; R ) we can apply Corollary 1 and Theorem 5 with r = ∞ and p, q = 2 to complete the proof of (74). Furthermore, using Theorem 5 with r = ∞ and p, q = 2 and ℓ = 1 completes the proof of (75).

and

Building upon Lemma 5 and starting from (58), we can derive an upper bound for L dec :

where (a) follows from the definition of h ( t ) , (b) is due to the fact that h ( t ) ⩽ ∥ h ∥ L ∞ (Ω; R ) for t ∈ Ω , (c) follows by applying Lemma 5, and (d) is a result of applying the Cauchy-Schwartz inequality.

B.1 Proof of Theorem 1

Proof. As previously mentioned, u dec ( · ) is a second-order smoothing spline fitted on the data points { ( β i 1 , f ( u enc ( β i 1 )) , . . . , ( β i |F| , f ( u enc ( β i |F| ))) } , where F := { β i 1 , . . . , β i |F| } represents the set of non-straggler worker nodes, and f ◦ u enc := { f ( u enc ( β i 1 )) , . . . , f ( u enc ( β i |F| )) } is the corresponding set of computation results from these non-straggler workers. By the definition given in (50), S λ d , |F| , 2 ( · ) denotes the smoothing spline operator for the decoder layer. Hence, we have the following:

where f = { f ( u enc ( β i 1 )) , . . . , f ( u enc ( β i |F| )) } . Let us define the following variables analogous to those in (6):

Since there are at most S stragglers among the worker nodes, we have ∆ F max ⩽ ( S +1) · ∆ max and ∆ F max ∆ F min ⩽ ( S +1) · ∆ max ∆ min ⩽ ( S +1) B . Additionally, because ∆ min ⩽ 2 N , there exists a constant J such that ∆ max ⩽ J N , and consequently, ∆ F max ⩽ J ( S +1) N ⩽ J ( S +1) N -S . Otherwise, this would contradict the condition ∆ max ∆ min ⩽ B .

Applying Theorem 6 with Ω = ( -1 , 1) , m = 2 , we have and

where L = p 2 ( ∆ F max ∆ F min ) · ( N -S )∆ F max 4 λ d + D (2) · ( ∆ F max ) 4 and H 0 , H 1 := H (2 , 0) , H (2 , 1) as defined in Theorem 6. Thus, we have:

where D := D (2) and (a) and (b) follow from ∆ F max ⩽ J ( S +1) N -S and ∆ F max ⩽ J ( S +1) N respectively. Substituting in (91),(92), and (92), we have:

Therefore, we can derive an upper bound for (77) based on the above inequalities. This upper bound holds for any λ d. Since λ d ⩽ N -4 and ∆ F max ∆ F min ⩽ ( S +1) B , we have:

where ˜ p 3 is a degree-3 polynomial in ( S +1) with positive constant coefficients, and ˜ D is the sum of the coefficients of ˜ p 3 and DJ 4 . Therefore, we have:

where r ( S, N ) := ˜ D ( S +1) 4 N 4 . Note that, since S +1 ⩽ N then 1 + r ( S,N ) 16 ⩽ 2 max(1 , ˜ D ) . Defining η := 2 max(1 , ˜ D ) , we have:

Defining C := H 1 2 0 H 1 2 1 η 1 4 and applying Lemma 2, completes the proof.

B.2 Proof of Theorem 2

Using the decomposition (56) we have:

where f = { f ( u enc ( β i 1 )) , . . . , f ( u enc ( β i |F| )) } and ϵ = { ϵ i 1 , . . . , ϵ i |F| } . Same as (79) we define the following variables:

Again we have ∆ F max ⩽ ( S +1) · ∆ max and ∆ F max ∆ F min ⩽ ( S +1) · ∆ max ∆ min (a) ⩽ ( S +1) B , where (a) is because of the bounded condition that we have. Therefore, ∆ F max ∆ F min is bounded. Additionally, since ∆ min ⩽ 2 N ,

then ∆ max ∆ min ⩽ B implies that both ∆ max = O ( 1 N ) . Thus, there exists a constant J such that ∆ max ⩽ J N . Therefore, we have ∆ F max ⩽ ∆ max · ( S +1) ⩽ J ( S +1) N ⩽ J ( S +1) N -S . Applying Theorems 6 and 7 with Ω = ( -1 , 1) , m = 2 , we have:

and

where L = p 2 ( ∆ F max ∆ F min ) · ( N -S )∆ F max 4 λ d + D (2) · ( ∆ F max ) 4 , H 0 , H 1 := H (2 , 0) , H (2 , 1) , and Q 0 ( λ 0 ) , Q 1 ( λ 0 ) := Q (2 , 0 , λ 0 ) , Q (2 , 1 , λ 0 ) as defined in Theorems 6 and 7. Since ∆ F max ∆ F min ⩽ ( S +1) B , we have:

where D := D (2) , ˜ p 2 ( S +1) = p 2 ( B ( S +1)) , p 3 ( S +1) := ˜ p 2 ( S +1) · J ( S +1) 4 is a degree three polynomial of ( S +1) , and (a) follows from the ∆ F max ⩽ J ( S +1) N -S . Substituting in (91), we have:

where (a) follows from the fact that λ -1 d ( N -S ) -4 ⩽ 1 , as assumed in Theorem 7 and (b) is by definition p 4 ( S + 1) := p 3 ( S + 1) + DJ 4 ( S + 1) 4 is a degree four polynomial of ( S + 1) . An analogous upper bound can be derived for E ϵ, F [ ∥ h ′ ∥ 2 L 2 (Ω; R ) ] as

where (a) follows from the definition λ d ⩽ λ 0 , (b) is derived from the definition ˜ η := H 1 ( 1 + λ 0 p 4 ( S +1) 16 ) 1 2 . By applying the upper bound for L dec from (77) and incorporating the

results from (94) and (95), we can deduce the following:

where (a) follows by the definitions µ 0 ( S ) := H 0 · p 4 ( S +1) and µ 1 ( S ) := ˜ η · p 4 ( S +1) 1 2 , and (b) is due to Q 0 ( λ 0 ) , Q 1 ( λ 0 ) ⩽ Q max ( λ 0 ) := max { Q 0 ( λ 0 ) , Q 1 ( λ 0 ) } and µ 0 ( S ) , µ 1 ( S ) ⩽ µ max ( S ) := max { µ 0 ( S ) , µ 1 ( S ) } . Therefore, we can conclude that:

Based on the definition of µ max ( S ) we have:

˜ where H max := max { H 0 , H 1 } and ˜ p 4 ( S ) := p 4 ( S ) 1 2 · ( 1+ p 4 ( S ) 16 ) 1 2 . Based on definition of Q max ( λ 0 ) (mentioned in Theorem 7), we have:

where w max := max { w 0 , w 1 } and ˜ w max := max { ˜ w 0 , ˜ w 1 } . Therefore we have:

Since (97) holds for all λ d, by minimizing the right-hand side of (97) with respect to λ d, we obtain:

By substituting the expression for λ ∗ d into (97), we have:

Thus, defining C 2 := 3 H max 4 w max , C ( λ 0 ) :=   max { 4 ,λ 0 } 1 2 max { λ 1 4 0 , ˜ w max w max }   , and using previously driven upper bound for L enc in Lemma 2 completes the proof.

B.3 Proof of Theorem 3

The upper bounds presented in Theorems 1 and 2 depend on ∥ ∥ ( f ◦ u enc ) (2) ∥ ∥ 2 L 2 (Ω; R ) , with exponents of 2 5 and 1 , respectively. By applying the chain rule, we can demonstrate that:

˜ where (a) follows from the chain rule, (b) is derived using the Cauchy-Schwartz inequality, (c) is due to ∥ f ′′ 0 ∥ L ∞ (Ω; R ) ⩽ ν , (d) follows from Theorem 5 with r = 4 , p = q = 2 , l = 1 , (e) follows from AM-GM inequality, (f) is a result of adding positive terms ∥ u enc ∥ 2 W 2 , 2 (Ω; R ) and ∥ u ′ enc ∥ 2 W 2 , 2 (Ω; R ) to the first term and ∥ u enc ∥ 2 W 2 , 2 (Ω; R ) to the second term in the parenthesis, (g) is result of applying Corollary 4, (h) is by defining ψ ( t ) := 73 t +4 × 73 2 t 2 , and (i) is because of Proposition 2.

Combining (103) with Theorems 1 and 2 we have

for the noiseless setting and

for the noisy setting, where the parameters λ e and λ e are as follows:

Note that since ψ ( t ) and γ ( t ) := t 2 5 are monotonically increasing in R + , its composition is monotonically increasing as well. Moreover, λ e and ˜ λ e share the same exponent of N as in Theorems 1 and 2, respectively. Consequently, the provided upper bound does not compromise the convergence rate.

B.4 Proof of Proposition 1

For part (i), we know that for t ⩽ M , we have:

ψ ( t ) = 73 t +4 × 73 2 t 2 ⩽ t (73 + 4 × 73 2 t ) ⩽ t (73 + 4 × 73 2 · M ) = t ( m 1 + m 2 M ) , (108) where m 1 := 73 , m 2 := 4 × 73 2 . Therefore, if ∥ u enc ∥ 2 H 2 (Ω; R ) ⩽ M , then:

where (a) is because of ∥ u enc ∥ 2 H 2 (Ω; R ) ⩽ M . Thus, we have:

For part (ii), let ˜ u enc ( t ) be a natural spline used as the encoder function fitted to the data points { ( α k , x k ) } K k =1 . Then, we have:

where (a) follows from the optimality of u ∗ ( · ) , and (b) follows from ˜ u enc ( α k ) = x k for k ∈ [ K ] . Therefore, ∫ Ω | ( u ∗ ) ′′ ( t ) | 2 dt ⩽ ∫ Ω | u enc ′′ ( t ) | 2 dt .

˜ Since u ∗ ( · ) is smoothing spline, it has the representation in natural spline space (as mentioned in (51)):

where, { b k ( · ) } K k =1 is a basis functions of second order natural splines. Therefore, using CauchySchwartz inequality, we have:

and

Both (113) and (114) hold for all t ∈ Ω . Thus:

where ξ := [ ξ 1 , . . . , ξ K +4 ] T = ( N T N + λ Φ ) -1 N T x with N ij = b i ( α j ) , Φ ij = ∫ Ω b ′′ i ( t ) b ′′ j ( t ) dt for i, j ∈ [ K +4] as defined in (51), and x := [ x 1 , . . . , x K ] . Noted that {∥ b ′ k ∥ 2 L ∞ (Ω; R ) } K +4 k =1 and {∥ b k ∥ 2 L ∞ (Ω; R ) } K +4 k =1 depend only on { α k } K k =1 .

Lemma 6. If ˜ ξ := ( N T N ) -1 N T x , then ∥ ξ ∥ 2 2 ⩽ κ (Φ) · ∥ ∥ ∥ ˜ ξ ∥ ∥ ∥ 2 2 < 2 | det Φ | ( ∥ Φ ∥ 2 F K ) K 2 ∥ ∥ ∥ ˜ ξ ∥ ∥ ∥ 2 2 , where κ (Φ) is condition number of Φ .

Define z := ( ˜ N T ˜ N ) -1 ˜ N T x . Thus, ˜ N T x = ˜ N T ˜ Nz . Thus, by applying the Cauchy-Schwartz inequality, we have:

where (a) is because V is an unitary matrix and λ 1 , . . . , λ K are eigenvalues of ˜ N . Continuing from (117), we obtain:

Let us define x 0 := ( N T N ) -1 N T x . Thus, we have:

Therefore, we can bound the ∥ ξ ∥ Φ := √ ξ T Φ ξ :

Since Φ is symmetric, by Rayleigh-Ritz theorem we know that:

where λ Φ min , λ Φ max are minimum and maximum eigenvalues of Φ , and (a) is due to the fact that since Φ is kernel matrix of RKHS space and { b k ( · ) } K k =1 are basis functions, it is positive definite. Thus, we have:

Applying the bound for condition number introduce in [66], we can complete the proof:

where ∥·∥ F is the Frobenius norm.

Using Lemma 6, (115), and (116) we have:

where (a) follows by applying Lemma 6. Setting M equal to the right-hand side of Equation (126) completes the proof.

B.5 Proof of Theorem 4

Consider a natural spline ˜ u enc ( t ) as the encoder function fitted on the data points { ( α k , x k ) } K k =1 . Let u ∗ enc ( t ) denote the optimal encoder minimizing the upper bound in (10). Then, we have:

˜ ˜ where (a) follows from the optimality of u ∗ enc ( t ) , and (b) is due to the fact that ˜ u enc ( α k ) = x k , since ˜ u enc ( · ) is a natural spline. Note that g ( ∥ ˜ u enc ∥ 2 ˜ H 2 (Ω; R ) ) is independent of N and S , and depends only on α k and x k for k ∈ [ K ] . Additionally, based on the Theorem 3 and (106), λ e = O ( S 3 N -3 ) and λ e = O ( S 8 5 N -3 5 ) for the noiseless and noisy cases, respectively. Thus, the upper bound provided in (10) converges at most at the rate of O ( S 3 N -3 ) for the noiseless case and O ( S 8 5 N -3 5 ) for the noisy case.

C Comparison with Berrut Coded Computing

C.1 Convergence rate

The upper bound of the infinity norm for the estimation provided in [29] for the coded computing scheme with N workers and maximum of S stragglers is as follows:

if N -s is odd, and if N -s is even, where R = ( s +1)( s +3) π 2 4 . Since ∥·∥ L 2 (Ω; R ) is upper bounded by ∥·∥ L ∞ (Ω; R ) , we can directly derive a convergence rate for the squared L 2 (Ω; R ) -norm of the error as N increases:

∥ ∥ ∥ ∥ Compared to our results, the upper bound for LeTCC provided in Theorem 1, O ( S 3 N -3 ) , is less sensitive to the number of stragglers and converges faster with increasing N . Note that, since the ∥·∥ L 2 (Ω; R ) is upper bounded by ∥·∥ L ∞ (Ω; R ) , the statement above does not guarantee faster convergence of the proposed scheme compared to Berrut approach.

It should be noted that [29] does not analyze the noisy setting.

C.2 Computational complexity

From the experimental view, we compare the whole encoding and decoding time (on a single CPU machine) for LeTCC and BACC frameworks, as shown in the following table:

Table 2: Average and std of end-to-end processing time of LeTCC and BACC for different architectures

±

±

As shown in Table 2, the end-to-end processing time of the proposed framework is on par with BACC .

D Comparison with Lagrange Coded Computing

Although the only existing coded computing scheme for general functions is Berrut coded computing [29], with which we have compared our proposed scheme, other schemes are designed for specific computations, such as polynomial functions [3] and matrix multiplication [13]. To provide further comparison, we evaluate our proposed scheme against Lagrange coded computing ( LCC ) [3], which is specifically designed for polynomial computations, as follows:

D.1 Accuracy of function approximation

LCC is applicable only to polynomial computing functions [3]. Additionally, to enable recovery, the number of servers required must be at least ( K -1) × deg ( f ) + S + 1 worker nodes [3, 29]; otherwise, the master node cannot recover any results. Moreover, LCC is designed for computation over finite fields and encounters serious instability when computing over real numbers, particularly when ( K -1) × deg ( f ) is around 25 or higher [18, 29].

Wecompare the proposed framework with LCC in Figure 5. Note that if N < ( K -1) × deg ( f )+ S +1 , LCC cannot operate effectively. To adapt LCC for such cases, we approximate results by fitting a lower-degree polynomial to the available workers' outputs. We run LeTCC and LCC on the same set of input data and a fixed polynomial function for 20 trials, plotting the average performance and corresponding 95% confidence intervals in Figure 5. Figures 5a and 5b illustrate the performances of LCC and LeTCC for a low-degree polynomial and a small number of data points (deg ( f ) = 3 and K = 5 ). In contrast, Figures 5c and 5d show performance for a higher-degree polynomial and a larger dataset (deg ( f ) = 15 and K = 10 ). As shown in Figures 5a and 5b, LCC achieves exact results for S ≤ 7 . However, at larger values of S , as well as larger polynomial degree (as in Figures 5c and 5d), the proposed approach, without any parameter tuning, outperforms LCC in terms of both computational stability (lower variance) and recovery accuracy.

(a)

(b)

(c)

(d)

10

Polynomial degree 3, (N, K) = (20, 5)

LeTCC (Aenc = 0,Adec = 0)

Lagrange

0.0025

0.0020

Polynomial degree 3, (N, K) = (20, 5)

LeTCC (Aenc = 0,Adec = 0)

Lagrange

8

b

Polynomial degree 15, (N,K) = (20, 10)

LeTCC (Aenc = 0, Adec = 0)

Lagrange

30000

40000

Polynomial degree 15, (N, K) = (20, 10)

LeTCC (Aenc = 0,Agec = 0)

Lagrange

ellIf-AIl2

0.0015

ellIf-AII

20000

1

0.0010

]3

10000

Q

T

0.0005

0.0000

0

10000

0.0005

2

20000

0

2

Number of Stragglers

4

6

8

10

12

14

0

1

Z

Number of Stragglers

3

4

5

6

7

0.0

0.5

1.0

Number of Stragglers

1.52.02.5

O'E

3.5

4.0

30000

0

2

Number of Stragglers

4

6

8

D.2 Computational complexity

Encoding and decoding complexities in LCC are O ( N · log 2 ( K ) · log log( K ) · d ) and O (( N -S ) · log 2 (( N -S )) · log log(( N -S )) · m ) , respectively, where d and m are input and output dimensions of the computing function f ( · ) , respectively [3]. In contrast, as mentioned before, for smoothing splines, the encoding and decoding process, which involves evaluation on new points and calculating the fitted coefficients, have the computational complexity of O ( K.d ) and O (( N -s ) .m ) . Consequently, the computational complexity of the proposed scheme is less than LCC .

E Sensitivity Analysis

E.1 Sensitivity to number of stragglers

The smoothing parameters for each model show low sensitivity to the number of stragglers (or worker nodes). To find the optimal smoothing parameter, we use cross-validation across different S N values. The following table presents the optimal smoothing parameters for selected numbers of stragglers for LeNet5 with ( N,K ) = (100 , 60) and RepVGG with ( N,K ) = (60 , 20) , respectively. As shown in Table 3, the optimal values of λ e and λ d exhibit low sensitivity to the number of stragglers.

E.2 Sensitivity to smoothing parameters

To assess the performance of the proposed scheme with respect to the smoothing parameters, we vary each parameter individually around its optimal point while holding the other parameter fixed at their optimal value. We then record the average percentage increase in RMSE relative to the RMSE at the

optimal point. Figure 6 presents these results for LeNet with ( N,K,S ) = (100 , 60 , 20) (Figures 6a and 6b) and for RepVGG with ( N,K,S ) = (60 , 20 , 35) (Figures 6c and 6d).

(a)

(b)

(c)

(d)

e=入=10-8

Ad=入=10-4

0.20

入d=0

0.10

入e=0

0.08

0.15

0.06

RMSE

0.10

t00

0.05

0.02

0.00

0.00

6-

-8

7

-6

-5

-3

-2

12

-10

-8

6

log1o(Ad)

log10(Ae)

e=入= 10-5

Ad=入=10-3

入e=0

0.20

F0E'O

0.25

0.15

F070

E

0.10-

0.15

0.10

S0'0

0.05

Ad=0

00°0

0.00

8-

7

-6

S-

-4

-3

-2

-9

-8

7

-6

S-

-4

E-

-2

log10(A)

log1o(Ae)

As shown in Figure 6, the presence of more stragglers increases the sensitivity of LeTCC with respect to its smoothing parameter. However, even in a high-straggler regime, the RMSE increases by only around 3% when the smoothing parameter deviates from its optimal value by a scale of 10 .

F High-dimensional computing function

Let us consider more general cases where f = [ f 1 , . . . , f m ] is a vector-valued function, where each component function f j : R → R is q j -Lipschitz continuous. Based on (2), we have:

Let us define the following objective for the decoder function:

The solution to (131), denoted as u ⋆ dec , is a vector-valued function, where each component u dec j ( · ) is a smoothing spline function fitted to the data points { ( β v , f j ( u enc ( β v ))) } v ∈F . As a result, By defining q = √ ∑ m j =1 q 2 j and scaling up all upper bounds for L dec by a factor of m , all previous results and theorems seamlessly extend to high-dimensional computing functions.

G Coded data points

Figures 7b and 7c display coded samples generated by BACC and LeTCC , respectively, derived from the same initial data points depicted in Figure 7a. These samples are presented for the MNIST dataset with parameters ( N,K ) = (70 , 30) . From the figures, it is apparent (Specifically in paired ones that are shown with the same color) that while both schemes' coded samples are a weighted combination of multiple initial samples, BACC 's coded samples exhibit high-frequency noise. This observation suggests that LeTCC regression functions produce more refined coded samples without any disruptive noise.

(a) Initial inputs

(b)

BACC

coded samples

(c) LeTCC coded samples

99q64

9q64

?

7

7

77

4

8

2

b

9

9

9

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We detailed our contributions clearly in the abstract and the introduction sections of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:We provided all the details regarding the assumptions, conditions, and limitations in the framework explanations (Section 3), theorems (Section 4), as well as the experiments section (Section 5) in the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our paper includes theoretical results in Section 4. In our theorems, we clearly mentioned all the required assumptions, and a complete (and correct) proof of them is available in appendices (e.g., see Appendix B). Please see Section 3 for a full definition of the problem and introduction to the notations used in the paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We completely explained our proposed framework in Section 3 and we also provided details regarding our empirical evaluations in Section 3 in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We provided references to all the open datasets that we used in the paper. Regarding the code, we are happy to share it later if required.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provided full experimental details in the paper (see Section 5).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The details are provided in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provided all the details regarding our experiments in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We followed the NeurIPS code of ethics in our paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper is focused on developing a new framework for coded distributed computing and should be categorized as foundational research. We believe this work has no direct societal impact that should be explained in the paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is not applicable to our work and this paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve these.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve these.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on coded computing for machine learning and derives loss-minimizing encoding and decoding functions.\n\nSTRENGTHS:\nPlease see the “Questions” section.\n\nWEAKNESSES:\nPlease see the “Questions” section.\n\nQUESTIONS:\nMy review is as follows:\n\nMajor: \n\nThe biggest thing that confused me when reading the paper was the use case for this kind of method. As far as I understand, this method is specifically designed for inference. Inference of computer vision models such as the ones used in the experiments of this paper (VGG, VIT) is actually very fast in practice. Even mobile phones can run inference on these models within a few tens or hundreds of milliseconds. What application would require to run many inferences for these models in parallel in a timely manner? \n\nI understand that the results are interesting from a theoretical point of view. It’s nice that encoding and decoding functions can be derived theoretically. But from a practical perspective, I cannot think of a scenario where one would need to make their inference straggler-resilient because it is typically already very low latency. (Furthermore, if an even lower latency is needed, then quantization is usually the direction to explore.) Please let me know if I’m missing something here. Perhaps, straggler resilient computing makes more sense for large-scale computing (e.g. large-model training) where you have a large-scale computation that is distributed onto a compute cluster.\n\nMinor: \n\nSome of the related work on coded computing has the property that if a given number of nodes return their solution, the exact output can be recovered and if not, an approximate solution can be generated. As far as I understand, the proposed method cannot recover the exact result even if all nodes return their result successfully. Please correct me if I’m wrong about this point. I wonder if this method could be modified to have the exact recovery property.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of improving the reliability of distributed computing architectures by encoding the input data before it is processed by worker machines such that a good approximation of the desired output can be reconstructed using only a subset of the workers’ outputs. They utilize learning theory to determine a regularized objective function for the decoder as well as a loss function for the overall system, and provide solutions for the encoder and decoder which optimize their derived upper bounds on the loss function. The objective function is based on kernel ridge regression, which leads to a second-order smoothing spline solution for the decoder. Both the noiseless and the noisy cases are considered, and the proposed method is compared with the existing approach of Berrut coded computing in terms of its convergence rate in the noiseless setting and its empirical performance. To evaluate the empirical performance, distributed inference is performed for deep learning models of varying sizes and tasks with varying sizes for the output vector. The experimental results show that LETCC consistently achieves a lower estimation error on the model outputs.\n\nSTRENGTHS:\n# Originality\nThe main contribution which differentiates the LeTCC method from prior works like BACC appears to be the introduction of regularization which causes the optimal solution to be based on smoothing splines rather than interpolation. Existing papers which have considered optimizing cost functions only focus on learning how to generate parity data points. Overall, this paper brings a novel method to the table and adequately cites related works.\n# Quality\nAlthough I lack experience with some of the mathematical tools used, to my knowledge there is nothing incorrect about the technical results. The experimental methodology seems to allow for a fair comparison between LeTCC and BACC for the use case of image classification with a deep learning model. In terms of reconstruction accuracy, the experimental results validate that LeTCC is superior to BACC, in some cases by a significant margin.\n# Clarity\nThe main content of the paper is laid out clearly with some small typos but no major issues.\n# Significance\nThe method for coded computing introduced in this paper represents an advancement in the level of accuracy that can be achieved when the function applied by the worker nodes is a complex deep learning model. The authors show that this advancement can be achieved by adapting the encoder and decoder to the function through the tuning of the regularization weight, which is an interesting idea.\n\nWEAKNESSES:\n# Originality\nThis paper distinguishes itself sufficiently from existing works, so I see no issues in terms of originality.\n# Quality\nLeTCC is proven to have a reconstruction error that scales better than BACC in terms of the number of worker nodes, but its scaling in terms of the number of stragglers is not directly compared. Based on Theorem 9 in the BACC paper, it appears that the max error along a single dimensions scales as $S^2$. The mean-squared error (MSE) for LeTCC scales as $S^4$, but since the max norm is upper bounded by the Euclidean norm it is unclear how to compare these results. It would be nice to compare the theoretical dependence on the number of stragglers, especially since the experimental results are mixed (in two cases the difference in MSE shrinks as $S$ increases, and in the other case the MSE grows as $S$ increases). \n# Clarity\nThe authors should explicitly state how many trials were averaged over to produce the plots in figure 3 to help readers interpret and/or reproduce the results.\n\nI identified the following typos in the paper:\n- Line 502: There is a “loc” subscript missing\n- Line 49: The sentence starting on this line is grammatically incorrect, I presume that the second comma is supposed to be replaced with the word “is”\n- Figure 3: The title of the top-left plot seems to have the values of N and K reversed\n\n# Significance\nWithout more details on the hyperparameter optimization process and the relative computational complexity of fitting the encoder and decoder functions as compared to those of BACC, it is hard to say whether LeTCC is more practically advantageous to use than BACC. It could be that the overhead introduced by hyperparameter tuning outweighs the benefit from reduced reconstruction error in some cases. It would have been helpful to see how tolerant LeTCC is to changes in the smoothing parameters, i.e. how much variation can be allowed while still outperforming BACC. One can also see from Figure 3 that a large difference in the mean-squared error does not consistently translate to a large difference in relative accuracy (at least when deep models are used), which makes it less likely that LeTCC’s performance benefits would be worth the additional overhead. As it currently stands, these issues reduce the practical value of the contribution made by this paper.\n\nQUESTIONS:\nAccording to Theorem 4, the mean-square error of LeTCC has a much worse scaling in terms of the number of stragglers in the noiseless setting than in the noisy setting. Does this originate from a sacrifice made to improve the scaling with respect to the number of worker nodes, or is there some other cause?\n\nLIMITATIONS:\nThe authors are upfront about the limited scope of their work. However, as mentioned in the weaknesses section, the extra overhead introduced by the need for hyperparameter tuning is something that I would have liked to have seen addressed in the paper. As mentioned in the weaknesses section, the comparison of error convergence rates with BACC is also limited.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a learning theory-based novel framework for coded computing with a focus on distributed machine learning applications. The proposed method sends mixtures of input samples to the worker nodes that compute the desired results on the mixtures. An encoder and a decoder functions are fitted at the master node using input samples and the result from workers, respectively. Finally, the decoder function can be used to estimate the output of computing function for the input samples. It is shown that the loss function (divergence of the estimated output with the true output of computing functions) is upper bounded by the generalization error of the decoder regression and training error of the encoder regression under some conditions. Experimental evaluations are provided to show the efficacy of the proposed method over the state-of-the-art baseline.\n\nSTRENGTHS:\n1. The proposed method leverages learning theory for coded computing which seems novel and interesting.\n2. Rigorous theoretical analysis is provided including convergence rate derivation and recoverability analysis.\n\nWEAKNESSES:\n1. The experimental section does not look comprehensive enough for the following reasons:\n   (a) Only one baseline is considered.  \n   (b) Authors acknowledge in Section 3 that the computational efficiency of encoder and decoders is a crucial factor. Yet, this was not reported in the numerical section.\n\n2. The computing functions (which are generally pre-trained neural networks) are evaluated on (probably) mixtures of input samples at the worker nodes. Those neural networks were trained on the data distribution. However, the mixture of input samples might not fall on the support of data distribution. Although for images, this does not seem to be an issue, there could be unexpected behavior in general.\n\nQUESTIONS:\n1. Could the authors provide comparision with additional baselines?\n2. Could the authors report the computational resources needed for fitting and evaluating the encoder and decoder?\n3. Could the authors provide a discussion for the weaknesses number 2 explained above? Also, does this framework work (or can it be modified to work) for computing functions that take discrete inputs?\n\nLIMITATIONS:\nYes, but it is spread out throughout the manuscript. A separate limitation section/paragraph would be helpful for the readers.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with coded distributed computing. I need to note that it is very popular research area now with a vast number of papers. But the authors are right when mention that the majority of papers utilizes standard algebraic codes (such as Reed-Solomon codes). The main problem of such approach is that these codes are designed over finite fields, which is not the case for machine learning tasks. The improvements utilize real or complex valued codes (e.g. Reed-Solomon codes over real or complex fields) but they face with the problems of accuracy (Vandermonde matrices have bad condition numbers). Moreover such approaches work only for some particular functions, e.g. polynomial ones. In this paper the authors propose a framework, which outperforms the Berrut approximate coded computing (BACC), which is the state-of-the-art coded computing scheme for general computing.\n\nSTRENGTHS:\nThe main strong point are as follows:\n- new framework foundation for coded computing, new loss function and its upper bound by decomposition\n- theoretical analysis and guarantees. Under some assumptions the authors find the optimum encoding and decoding functions and characterize the convergence rate for the expected loss in both noise-free and noisy computations.\n- Numerical analysis, that shows the new framework to outperforms the Berrut approximate coded computing (BACC), which is the state-of-the-art coded computing scheme for general computing.\n\nWEAKNESSES:\nI list the main weaknesses below:\n- «We develop a new foundation for coded computing, based on the theory of learning, rather than the theory of coding». I would not be so categorical. Coding is a method when you add and the utilize the redundancy to deal with errors and erasures. The main advantage of your approach is that you are not using standard finite field codes.\n- How the optimal (under some assumptions) encoder and decoder functions in section 4 are related to the functions utilized for numerical experiments (DNNs)? Is it possible to analyse the performance under optimal encoder and decoder and compare it to the results from Section 5?\n- You made the comparison to BACC, which is a framework for general computing. Could you please make a comparison for some particular (e.g. polynomial) problems? I just wonder if proposed general approach is competitive with e.g. Lagrange computing and what you should pay for universality. So my claim is that more scenarios and computing problems should be checked to understand the limitations and applicability of your method.\n- How flexible is your approach if the system parameters change (the number of worker nodes or the number of stragglers).\n- Figure 3 seems to suffer from the lack of statistics (you need more experiments).\n- You mentioned Byzantine case. It is known to be a problem for the codes over the real field (in case of finite field the approaches are well-developed). It would be beneficial to briefly explain how you plan to deal with this problem.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nLimitations are well described.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}