## 4 The Proposed Collaborative Learning Method
### Theoretical Basis of the Collaborative Learning Method  
First, we show how to compute the optimal parameters of a linear regression model incrementally. Assume that the number of features is two. Thus,  
\[y=\beta_{0}+\beta_{1}x_{1}+\beta_{2}x_{2}.\]  
The optimal values of each \(\beta_{j}\), \(j=0,1,2\) can be obtained by finding the minimum value of the function \(L(\vec{\beta})\). Let  
\[L(\beta_{0},\beta_{1},\beta_{2})=\sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{1 }^{i}-\beta_{2}x_{2}^{i})^{2}.\]  
Solve \(\dfrac{\partial L}{\partial\beta_{i}}=0\), \(i=0,1,2\). We obtain:  
\[\sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{1}^{i}-\beta_{2}x_{2}^ {i}) =0\] \[\sum_{i=1}^{n}x_{1}^{i}(y_{i}-\beta_{0}-\beta_{1}x_{1}^{i}-\beta_{ 2}x_{2}^{i}) =0\] \[\sum_{i=1}^{n}x_{2}^{i}(y_{i}-\beta_{0}-\beta_{1}x_{1}^{i}-\beta_{ 2}x_{2}^{i}) =0\]  
These equations are also called normal equations. They can be rewritten as:  
\[\sum_{i=1}^{n}y_{i} =\beta_{0}n+\beta_{1}\sum_{i=1}^{n}x_{1}^{i}+\beta_{2}\sum_{i=1}^ {n}x_{2}^{i} \tag{1}\] \[\sum_{i=1}^{n}x_{1}^{i}y_{i} =\beta_{0}\sum_{i=1}^{n}x_{1}^{i}+\beta_{1}\sum_{i=1}^{n}(x_{1}^{ i})^{2}+\beta_{2}\sum_{i=1}^{n}x_{1}^{i}x_{2}^{i}\] \[\sum_{i=1}^{n}x_{2}^{i}y_{i} =\beta_{0}\sum_{i=1}^{n}x_{2}^{i}+\beta_{1}\sum_{i=1}^{n}x_{1}^{ i}x_{2}^{i}+\beta_{2}\sum_{i=1}^{n}(x_{2}^{i})^{2}\]The coefficients \(\beta_{0},\beta_{1},\beta_{2}\) can be computed by solving systems of linear Equations (1). Now suppose another set of \(k\) data points are to be added:  
\[(x_{1}^{n+1},x_{2}^{n+1},y_{n+1}),\ldots,(x_{1}^{n+k},x_{2}^{n+k},y_{n+k}).\]  
Equations (1) become  
\[\sum_{i=1}^{n+k}x_{2}^{i} =\beta_{0}(n+k)+\beta_{1}\sum_{i=1}^{n+k}x_{1}^{i}+\beta_{2}\sum_ {i=1}^{n+k}x_{2}^{i}\] \[L(\beta_{0},\beta_{1},\beta_{2})=\sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{1 }^{i}-\beta_{2}x_{2}^{i})^{2}.\]0 \[L(\beta_{0},\beta_{1},\beta_{2})=\sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{1 }^{i}-\beta_{2}x_{2}^{i})^{2}.\]1  
The new sum can be expressed as the old sum plus extra terms. \[L(\beta_{0},\beta_{1},\beta_{2})=\sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{1 }^{i}-\beta_{2}x_{2}^{i})^{2}.\]2 \[L(\beta_{0},\beta_{1},\beta_{2})=\sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{1 }^{i}-\beta_{2}x_{2}^{i})^{2}.\]3 \[L(\beta_{0},\beta_{1},\beta_{2})=\sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_{1 }^{i}-\beta_{2}x_{2}^{i})^{2}.\]4  
and so on for the other summations. To generalize the method described above, it is more convenient to rewrite these equations by using matrices. \[ \begin{bmatrix}1&x_{1}^{1}&x_{2}^{1}&\ldots&x_{m}^{1}\\ 1&x_{1}^{2}&x_{2}^{2}&\ldots&x_{m}^{2}\\ \vdots&\vdots&\vdots&\ddots&\vdots\\ 1&x_{1}^{n}&x_{2}^{n}&\ldots&x_{m}^{n}\end{bmatrix}\begin{bmatrix}\beta_{0}\\ \beta_{1}\\ \vdots\\ \beta_{m}\end{bmatrix} = \begin{bmatrix}y_{1}\\ y_{2}\\ \vdots\\ y_{m}\end{bmatrix} .\]  
The relationship between matrix \(X\) and vector \(\vec{y}\) can also be written in matrix form as:  
\[X\vec{\beta}=\vec{y}. \tag{2}\]  
The loss function can then be rewritten as:  
\[L(\vec{\beta}) =||X\vec{\beta}-\vec{y}||^{2}\] \[=\Big{(}X\vec{\beta}-\vec{y}\Big{)}^{T}\Big{(}X\vec{\beta}-\vec{y }\Big{)}\  
Setting the gradient to 0, we get the optimum coefficients:  
\[-2X^{T}\vec{y}+2X^{T}X\vec{\beta}=0\] \[\Rightarrow X^{T}\vec{y}=X^{T}X\vec{\beta}\] \[\Rightarrow \vec{\beta}=\left(X^{T}X\right)^{-1}X^{T}\vec{y}.\]  
Therefore, the optimum value for \(\vec{\beta}\) is  
\[\vec{\beta}=\left(X^{T}X\right)^{-1}X^{T}\vec{y}. \tag{3}\]  
Suppose that each user \(u\) has a set of \(\left(x_{1}^{i},x_{2}^{i},\ldots,x_{m}^{i}\right)\) and its response \(y_{i}\). They want to collaboratively compute the optimal coefficients  
\[\vec{\beta}=\left(\beta_{0},\beta_{1},\ldots,\beta_{m}\right)\]  
such that \(||X\vec{\beta}-\vec{y}||^{2}\) is minimized. Assume that there are two parties: \(A\) and \(B\). \(A\)'s data are in the upper part of \(X\) and \(B\)'s data are in the lower part of \(X\). Matrix \(X\) can be written as \( \begin{bmatrix}X_{1}\\ X_{2}\end{bmatrix} \). Similarly, vector \(\vec{y}\) can be written as \(