SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
p _ {3} ^ {*} \leq p _ {r} \leq p _ {3} ^ {*} \left(1 + \frac {\sqrt {m _ {1} m _ {2}} R \sigma_ {r + 1}}{\beta}\right) ^ {2}, \tag {10}
$$
where  $p_r$  denotes the objective value achieved by the parameters trained using  $\mathbf{X}_r$ .
Remark 2. Theorem 1 and 2 imply that for a given arbitrary rank data matrix  $\mathbf{X}$ , the regularized training problem in (4) can be approximately solved by convex solvers to achieve a worst-case approximation  $p_3^* \left(1 + \frac{\sqrt{m_1 m_2} R \sigma_{r+1}}{\beta}\right)^2$  with complexity  $\mathcal{O}(d^3 m_1^3 2^{3(m_1 + 1)} n^{3(m_1 + 1)r})$ , where  $r \ll \min\{n, d\}$ . Therefore, even for full rank data matrices where the complexity is exponential in  $n$  or  $d$ , one can approximately solve the convex program in (8) in polynomial-time. Moreover, we remark that the approximation error proved in Theorem 2 can be arbitrarily small for practically relevant problems. As an example, consider a parallel network training problem with  $\ell_2$  loss function, then the upperbound becomes  $(1 + \frac{\sqrt{m_1 m_2} \sigma_{r+1}}{\beta})^2$ , which is typically close to one due to fast decaying singular values in practice (see Figure 4).
3.2 Representational power: Two versus three layers
Here, we provide a complete explanation for the representational power of three-layer networks by comparing with the two-layer results in [6]. We first note that three-layer networks have substantially higher expressive power due to the non-convex interactions between hidden layers as detailed in [39, 40]. Furthermore, [41] show that layerwise training of three-layer networks can achieve comparable performance to deeper models, e.g., VGG-11, onImagenet. There exist several studies analyzing two-layer networks, however, despite their empirical success, a full theoretical understanding and interpretation of three-layer networks is still lacking in the literature. In this work, we provide a complete characterization for three-layer networks through the lens of convex optimization theory. To understand their expressive power, we compare our convex program for three-layer networks in (8) with its two-layer counterpart in [6].
[6] analyzes two-layer networks with one ReLU layer, so that the data matrix  $\mathbf{X}$  is multiplied with a single diagonal matrix (or hyperplane arrangement)  $\mathbf{D}_i$ . Thus, the effective data matrix is in the form of  $\tilde{\mathbf{X}}_s = [\mathbf{D}_1\mathbf{X}\dots \mathbf{D}_P\mathbf{X}]$ . However, since our convex program in (8) has two nonlinear ReLU layers, the composition of these two-layer can generate substantially more complex features via locally linear variables  $\{\mathbf{w}_{ij_1l}^s\}$  multiplying the  $d$ -dimensional blocks of the columns of the effective data matrix  $\tilde{\mathbf{X}}_s$  in Theorem 1. Although this may seem similar to the features in [21], here, we have  $2^{m_1}$  variables for each linear region unlike [21] which employ 2 variables per linear region. Moreover, [21] only considers the case where the second hidden layer has only one neuron, i.e.,  $m_2 = 1$ , which is not a standard deep networks. Hence, we exactly describe the impact of having one more ReLU layer and its contribution to the representational power of the network.
In this section, we present numerical experiments corroborating our theory.
Low rank model in Theorem 2: To validate our claims, we generate a synthetic dataset as follows. We first randomly generate a set of layer weights for a parallel ReLU network with  $K = 5$  subnetworks by sampling from a standard normal distribution. We then obtain the labels as  $\mathbf{y} = \sum_{k}\left((\mathbf{X}\mathbf{W}_{1k})_{+}\mathbf{W}_{2k}\right)_{+}\mathbf{w}_{3k} + 0.1\epsilon$ , where  $\epsilon \sim N(\mathbf{0},\mathbf{I}_n)$ . To promote a low rank structure in the data, we first sample a matrix from the standard normal distribution and then set  $\sigma_{r + 1} = \ldots = \sigma_d = 1$ . We consider a regression framework with  $\ell_2$  loss and  $(n,r,\beta ,m_1,m_2) = (15,d / 2,0.1,1,1)$  and present the numerical results in Figure 4. Here, we observe that the low rank approximation of the objective  $p_r$  is closer to  $p_3^*$  than the worst-case upper-bound predicted by Theorem 2. However, in Figure 4b, the low rank approximation provides a significant reduction in the number of hyperplane arrangements, and therefore in the complexity of solving the convex program.
Toy dataset: We use a toy dataset with 5 samples and 2 features, i.e.,  $(n,d) = (5,2)$ . To generate the dataset, we forward propagate i.i.d. samples from a standard normal distribution, i.e.,  $\mathbf{x}_i\sim \mathcal{N}(\mathbf{0},\mathbf{I}_d)$ , through a parallel network with 3 layers, 5 sub-networks, and 3 neurons, i.e.,  $(L,K,m_1,m_2) = (3,5,3,1)$ . We then train the parallel network in (4) on this toy dataset using both our convex program (8) and non-convex SGD. We provide the training objective and wall-clock time in Table 2, where we particularly include 5 initialization trials for SGD. This experiment shows that when the number of sub-networks  $K$  is small, SGD trials fail to converge to the global minimum achieved by our convex program. However, as we increase  $K$ , the number of trials converging to global minimum gradually increases. Therefore, we show the benign overparameterization impact.
Image classification: We conduct experiments on benchmark image datasets, namely CIFAR-10 [42] and Fashion-MNIST [43]. We particularly consider a ten class classification task and use a parallel network with 40 sub-networks and 100 hidden neurons, i.e.,  $(K,m_{1},m_{2}) = (40,100,1)$ . In Figure 5, we plot the test accuracies against wall-clock time, where we include several different optimizers as well as SGD. Moreover, we include a parallel network trained with SGD/Adam and Weight Decay (WD) regularization to show the effectiveness of path regularization in (4). We first note that our convex approach achieves both faster convergence and higher final test accuracies for both dataset. However, the performance gain for Fashion-MNIST seems to be significantly less compared to the CIFAR-10 experiment. This is due to the nature of these datasets. More specifically, since CIFAR-10 is a much more challenging dataset, the baseline accuracies are quite low (around  $\sim 50\%$ ) unlike Fashion-MNIST with the baseline accuracies around  $\sim 90\%$ . Therefore, the accuracy improvement achieved by the convex program seems low in Figure 5b. We also observe that weight decay achieves faster convergence rates however path regularization yields higher final test accuracies. It is normal to have faster convergence with weight decay since it can be incorporated into gradient-based updates without any computational overhead.
Parallel neural networks were previously investigated by [22, 23]. Although these studies provided insights into the solutions, they require impractical assumptions, e.g., sparsity among sub-networks in Theorem 1 of [23]) and linear activations and hinge loss assumptions in [22].
Recently, [6] studied weight decay regularized two-layer ReLU network training problems and introduced polynomial-time trainable convex formulations. However, their analysis is restricted to standard two-layer ReLU networks, i.e., in the form of  $f_{\theta}(\mathbf{X}) = (\mathbf{X}\mathbf{W}_1)_+ \mathbf{w}_2$ . The reasons for this restriction is that handling more than one ReLU layer is a substantially more challenging optimization problem. As an example, a direct extension of [6] to three-layer NNs will yield doubly exponential complexity, i.e.,  $\mathcal{O}(n^{rn^r})$  for a rank-  $r$  data matrix, due to the combinatorial behavior of multiple ReLU layers. Thus, they only examined the case with a single ReLU layer (see Table 1 for details and references in [6]). Additionally, since they only considered standard two-layer ReLU networks, their analysis is not valid for a broader range of NN-based architectures as detailed in Remark 1. Later on, [21] extended this approach to three-layer ReLU networks. However, since they analyzed  $\ell_2$ -norm regularized training problem, they had to put unit Frobenius norm constraints on
4 Details on the experiments can be found in Appendix A.1.
(b) Fashion-MNIST
Figure 5: Accuracy of a three-layer architecture trained using the non-convex formulation (4) and the proposed convex program (8), where we use (a) CIFAR-10 with  $(n,d,m_1,m_2,K,\beta,\text{batch size}) = (5\times 10^4,3072,100,1,40,10^{-3},10^3)$  and (b) Fashion-MNIST with  $(n,d,m_1,m_2,K,\beta,\text{batch size}) = (6\times 10^4,784,100,1,40,10^{-3},10^3)$ . We note that the convex model is trained using (a) SGD and (b) Adam.
the first layer weights, which does not reflect the settings in practice. In addition, their analysis is restricted to the networks with a single neuron in the second layer (i.e.  $m_2 = 1$ ) that is in the form of  $f_{\theta}(\mathbf{X}) = \sum_{k}\left((\mathbf{X}\mathbf{W}_{1k})_{+}\mathbf{w}_{2k}\right)_{+}w_{3k}$ . Since this architecture only allows a single neuron in the second layer, each sub-network  $k$  has an expressive power that is equivalent to a standard two-layer network rather than three-layer. This can also be realized from the definition of the constraint set  $\mathcal{C}$  in Theorem 1. Specifically, the convex set  $\mathcal{C}$  in [21] has decoupled constraints across the hidden layer index  $j_1$  whereas our formulation sums the responses over hidden neurons before feeding through the next layer as standard deep networks do. Therefore, this analysis does not reflect the true power of deep networks with  $L > 2$ . Moreover, the approach in [21] has exponential complexity when the data matrix has full rank, which is unavoidable.
However, we analyze deep neural networks in (1) without any assumption on the weights. Furthermore, we develop an algorithm which has fully polynomial-time complexity in data dimensions and prove strong global optimality guarantees for this algorithm in Theorem 2.
We studied the training problem of path regularized deep parallel ReLU networks, which includes ResNets and standard deep ReLU networks as its special cases. We first showed that the non-convex training problem can be equivalently cast as a single convex optimization problem. Therefore, we achieved the following advantages over the training on the original non-convex formulation: 1) Since our model is convex, it can be globally optimized via standard convex solvers whereas the non-convex formulation trained with optimizers such as SGD might be stuck at a local minimum, 2) Thanks to convexity, our model does not require any sort of heuristics and additional tricks such as learning rate schedule and initialization scheme selection or dropout. More importantly, we proposed an approximation to the convex program to enable fully polynomial-time training in terms of the number of data samples  $n$  and feature dimension  $d$ . Thus, we proved the polynomial-time trainability of deep ReLU networks without requiring any impractical assumptions unlike [6, 21]. Notice that we derive an exact convex program only for three-layer networks, however, recently [19] proved that strong duality holds for arbitrarily deep parallel networks. Therefore, a similar analysis can be extended to deeper networks to achieve an equivalent convex program, which is quite promising for future work. Additionally, although we analyzed fully connected networks in this paper, our approach can be directly extended to various NN architectures, e.g., threshold/binary networks [44], convolution networks [45], generative adversarial networks [46], NNs with batch normalization [47], autoregressive models [48], and Transformers [49, 50].
[57] Maurice Sion. On general minimax theorems. Pacific J. Math., 8(1):171-176, 1958.
[58] Piyush C Ojha. Enumeration of linear threshold functions from the lattice of hyperplane intersections. IEEE Transactions on Neural Networks, 11(4):839-850, 2000.
[59] Richard P Stanley et al. An introduction to hyperplane arrangements. Geometric combinatorics, 13:389-496, 2004.
[60] RO Winder. Partitions of n-space by hyperplanes. SIAM Journal on Applied Mathematics, 14(4):811-818, 1966.
[61] Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers, (3):326-334, 1965.
[62] Arda Sahiner, Tolga Ergen, John M. Pauly, and Mert Pilanci. Vector-output relu neural network problems are copositive programs: Convex analysis of two layer networks and polynomial-time algorithms. In International Conference on Learning Representations, 2021.
A Appendix 15
A.1 Additional numerical results and details 15
A.2 Parallel ReLU networks 17
A.3 Proof of Lemma 1 18
A.4 Proof of Theorem 1 18
A.5 Proof of Proposition 1 20
A.6 Proof of Theorem 2 21
A.7 Proof for the dual problem in (3) 23
A.8 Hyperplane arrangements 23
A.9 Low rank model in Theorem 2 24
A.10 Proof of Proposition 2 and Corollary 1 24
A.11 Extension to vector outputs 26
A.1 Additional numerical results and details
In this section, we provide new numerical results and detailed information about our experiments in the main paper.
Decision boundary plots in Figure 1: In order to visualize the capabilities of our convex training approach, we perform an experiment on the spiral dataset which is known to be challenging for 2-layer networks while 3-layer networks can readily interpolate the training data (i.e. exactly fit the training labels) [51]. As the baselines of our analysis, we include the two-layer convex training approach in [6] and recently introduced three-layer convex training approach (with weight decay regularization and several architectural and parametric assumptions) in [21]. As our experimental setup, we consider a binary classification task with  $\mathbf{y} \in \{+1, -1\}^n$  and squared loss. We choose  $(n, m_1, m_2, P_1, P_2, \beta) = (30, 5, 1, 11, 11, 1e - 4)$  and for [6] we use  $P = 50$  neurons/hyperplane arrangements. We also use CVPXY with MOSEK solver [52-54] to globally solve the convex programs. As demonstrated in Figure 1, baselines methods, especially [21], fit a function that is significantly different than the underlying spiral data distribution. This clearly shows that since [6] is restricted two-layer networks and [21] have multiple assumptions, i.e., unit Frobenius norm constraints on the layer weights ( $\| \mathbf{W}_{lk} \|_F \leq 1$ ,  $\forall l \in [L - 2]$ ) and last hidden layer weights cannot be matrices ( $\mathbf{w}_{(L - 1)k} \in \mathbb{R}^{m_L - 2}$ ), both baseline approaches fail to reflect true expressive power of deep networks ( $L > 2$ ). On the other hand, our convex training approach for path regularized networks fits a model that successfully describes the underlying data distribution for this challenging task.
Additional experiments: We also conduct experiments on several datasets available in UCI Machine Learning Repository [55], where we particularly selected the datasets from [56] such that  $n \leq 500$ . For these datasets, we consider a conventional binary classification framework with  $(m_1, m_2, K, \beta) = (100, 1, 40, 0.5)$  and compare the test accuracies of non-convex architectures trained with SGD and Adam with their convex counter parts in (8). For these experiments, we use the  $80\% - 20\%$  splitting ratio for the training and test sets. Furthermore, we train each algorithm long enough to reach training accuracy one. As shown in Table 3, our convex approach achieves higher or the same test accuracy compared to the standard non-convex training approach for most of the datasets (precisely 20 and 19 out of 21 datasets for SGD and Adam, respectively). We also note that for this experiment, we used the unconstrained form in (11) with the approximate version in Remark 3.3 of [6].
Details for the experiments in Table 2 and Figure 5: We first note that for the experiments in Table 2, we use CVX/CVPXY [52-54] to globally solve the proposed convex program in (8). For these
Table 3: Test accuracies for UCI experiments  $((m_1, m_2, K, \beta) = (100, 1, 40, 0.5)$  and  $80\% - 20\%$  training-test split). Here, we present the standard non-convex architectures and the proposed convex counterpart trained with SGD and Adam optimizers. If one approach achieves higher accuracy on a certain dataset, we display the corresponding accuracy value in bold font. We observe that our convex approach achieves either higher or the same accuracy for 20 and 19 datasets (out of 21 datasets) when trained with SGD and Adam, respectively
SGD	Adam
Dataset	n	d	Non-convex	Convex(Ours)	Non-convex	Convex(Ours)
acute-inflammation	120	6	1.000	1.000	1.000	1.000
acute-nephritis	120	6	1.000	1.000	1.000	1.000
balloons	16	4	1.000	1.000	1.000	1.000
breast-cancer	286	9	0.690	0.707	0.655	0.672
breast-cancer-wisc-prog	198	33	0.750	0.800	0.800	0.825
congressional-voting	435	16	0.667	0.667	0.551	0.597
conn-bench-sonar-mines-rocks	208	60	0.714	0.786	0.738	0.833
echocardiogram	131	10	0.704	0.704	0.666	0.703
fertility	100	9	0.750	0.800	0.800	0.800
haberman-survival	306	3	0.710	0.774	0.677	0.709
heart-hungarian	294	12	0.831	0.831	0.779	0.813
hepatitis	155	19	0.645	0.710	0.709	0.741
ionosphere	351	33	0.887	0.901	0.929	0.887
molec-biol-promoter	106	57	0.818	0.818	0.727	0.772
musk-1	476	166	0.958	0.927	0.947	0.927
parkinsons	195	22	0.974	1.000	0.923	1.000
pittsburg-bridges-T-OR-D	102	7	0.952	0.952	0.809	0.857
planning	182	12	0.541	0.568	0.649	0.703
statlog-heart	270	13	0.759	0.796	0.759	0.833
trains	10	29	1.000	1.000	1.000	1.000
vertebral-column-2classes	310	6	0.806	0.839	0.758	0.822
Highest test accuracy
20/21 19/21
experiments, we use a laptop with i7 processor and 16GB of RAM. In order to tune the learning rate of SGD/GD, we first perform training with a bunch of learning different learning rates and select the one with the best performance on the validation datasets, which is 0.005 in these experiments.
For comparatively larger scale image classification experiments in Figure 5, we utilize a cluster GPU with 50GB of memory. However since the equivalent convex program in (8) has constraint which are challenging to handle for these datasets, we propose the following unconstrained convex problem which has the same global minima with the constrained version as discussed in [48]
$$
\min  _ {\mathbf {z}, \mathbf {z} ^ {\prime} \in \mathbb {R} ^ {d m _ {1} M P _ {1} P _ {2}}} \mathcal {L} (\tilde {\mathbf {X}} (\mathbf {z} ^ {\prime} - \mathbf {z}), \mathbf {y}) + \beta (\| \mathbf {z} \| _ {G, 1} + \| \mathbf {z} ^ {\prime} \| _ {G, 1}) + \lambda (h _ {\mathcal {C}} (\mathbf {z}) + h _ {\mathcal {C}} (\mathbf {z} ^ {\prime})) \tag {11}
$$
where  $\lambda > 0$  coefficient to penalize the violated constraints and  $h_{\mathcal{C}}(\mathbf{z})$  is a function to sum the absolute value of all constraint violations defined as
$$
h _ {\mathcal {C}} (\mathbf {z}) := \mathbf {1} ^ {T} \sum_ {i, j _ {1}, l, s} \left(\left(- (2 \mathbf {D} _ {1 i j _ {1}} - \mathbf {I} _ {n}) \mathbf {X} \mathbf {z} _ {i j _ {1} l} ^ {s}\right) _ {+}\right) + \mathbf {1} ^ {T} \sum_ {i, l, s} \left(- \sum_ {j _ {1}} \mathcal {I} _ {i j _ {1} l} ^ {s} (2 \mathbf {D} _ {2 l} - \mathbf {I} _ {n}) \mathbf {D} _ {1 i j _ {1}} \mathbf {X} \mathbf {z} _ {i j _ {1} l} ^ {s}\right) _ {+}
$$
Thus, we obtain an unconstrained version of convex optimization problem (11), where one can use commonly employed first-order gradient based optimizers, e.g., SGD and Adam, available in deep learning libraries such PyTorch and Tensorflow. For both CIFAR-10 and Fashion-MNIST, we use the same training and test splits in the original datasets. We again perform a grid search to tune the learning rate, where the best performance is achieved by the following choices
$$
\begin{array}{l} (\mu_ {C o n v e x}, \mu_ {S G D}, \mu_ {A d a m}, \mu_ {A d a g r a d}, \mu_ {A d a d e l t a}, \mu_ {S G D} ^ {W D}) = (5 e - 7, 5 e - 3, 2 e - 5, 2 e - 3, 3 e - 1, 1) \\ \text {a n d} \end{array}
$$
$$
\left(\mu_ {C o n v e x}, \mu_ {S G D}, \mu_ {A d a m}, \mu_ {A d a g r a d}, \mu_ {A d a d e l t a}, \mu_ {S G D} ^ {W D}\right) = (1 e - 5, 2 e - 1, 2 e - 3, 1 e - 2, 3, 1)
$$
as the learning rates for CIFAR-10 and Fashion-MNIST, respectively. We choose the momentum coefficient of the SGD optimizer as 0.9. In addition to this, we set  $P_{1}P_{2} = K$  and  $\lambda = 1e - 5$ .
Figure 6: Test accuracy for convex layerwise training, where each stage is our three-layer convex formulation in Theorem 1. Here, we train three-layer neural networks sequentially using convex optimization to build deep networks.
More importantly, we remark that these experiments are performed by using a small sampled subset of hyperplane arrangements rather than sampling all possible arrangements as detailed in Remark 3.3 of [6]. In particular, we first generate random weight matrices from a multivariate standard normal distribution and then solve the convex program using only the arrangements of the sampled weight matrices.
Details for the experiments in Figure 6: Layerwise training with shallow neural networks was proven to work remarkably well. Particularly, [41] shows that one can train three-layer neural networks sequentially to build deep networks that outperform end-to-end training with SOTA architectures. In Figure 6, we apply this layerwise training procedure with our convex training approach. In particular, each stage in this figure is our three-layer convex formulation in Theorem 1. Here, we use the same experimental setting in the previous section. We observe that making the network deeper by stacking convex layers resulted in significant performance improvements. Specifically, at the fifth stage, we achieved almost  $85\%$  accuracy for CIFAR-10 unlike below  $60\%$  accuracies in Figure 5.
A.2 Parallel ReLU networks
The parallel networks  $f_{\theta}(\mathbf{X})$  models a wide range of NNs in practice. As an example, standard NNs and ResNets [33] are special cases of this network architecture. To illustrate this, let us consider a parallel ReLU with two sub-networks and four layers, i.e.,  $K = 2$  and  $L = 4$ . If we set  $\mathbf{W}_{lk} = \mathbf{W}_l \forall k \in [2], l \in [4]$  then our architecture reduces to a standard four-layer network
$$
\sum_ {k = 1} ^ {2} \left(\left(\left(\mathbf {X W} _ {1 k}\right) _ {+} \mathbf {W} _ {2 k}\right) _ {+} \mathbf {W} _ {3 k}\right) _ {+} \mathbf {w} _ {4 k} = \left(\left(\left(\mathbf {X W} _ {1}\right) _ {+} \mathbf {W} _ {2}\right) _ {+} \mathbf {W} _ {3}\right) _ {+} \mathbf {w} _ {4},
$$
where  $\mathbf{W}_1\in \mathbb{R}^{d\times m_1}$ ,  $\mathbf{W}_2\in \mathbb{R}^{m_1\times m_2}$ ,  $\mathbf{W}_3\in \mathbb{R}^{m_2\times m_3}$ ,  $\mathbf{w}_4\in \mathbb{R}^{m_3}$ . For ResNets, we first remark that since residual blocks are usually used after a ReLU activation function, which is positively homogeneous of degree one, in practice, each residual block takes only nonnegative entries as its inputs. Thus, we can assume  $\mathbf{X}\in \mathbb{R}_{+}^{n\times d}$  without loss of generality. We also assume that weights obey the following form:  $\mathbf{W}_{11} = \mathbf{W}_1$ ,  $\mathbf{W}_{21} = \mathbf{W}_2$ ,  $\mathbf{W}_{12} = \mathbf{W}_{22} = \mathbf{I}_d$ ,  $\mathbf{W}_{31} = \mathbf{W}_{32} = \mathbf{W}_3$ , and  $\mathbf{w}_{41} = \mathbf{w}_{42} = \mathbf{w}_4$  then
$$
f _ {\theta} (\mathbf {X}) = \sum_ {k = 1} ^ {2} \left(\left(\left(\mathbf {X W} _ {1 k}\right) _ {+} \mathbf {W} _ {2 k}\right) _ {+} \mathbf {W} _ {3 k}\right) _ {+} \mathbf {w} _ {4 k} = \left(\left(\left(\mathbf {X W} _ {1}\right) _ {+} \mathbf {W} _ {2}\right) _ {+} \mathbf {W} _ {3}\right) _ {+} \mathbf {w} _ {4} + \left(\mathbf {X W} _ {3}\right) _ {+} \mathbf {w} _ {4}
$$
which is a shallow ResNet as demonstrated in Figure 1 of [25].
Let us first define  $r_{kj_{L-1}} \coloneqq \sqrt{\sum_{j_1, \ldots, j_{L-2}} \left(\|\mathbf{w}_{1kj_1}\|_2^2 \prod_{l=2}^{L-1} w_{lkj_{l-1}j_l}^2\right)} > 0$ . Notice that if  $r_{kj_{L-1}} = 0$ , this means that  $k^{th}$  sub-network does not contribute the output of the parallel network in (1). Therefore, we can remove the paths with  $r_{kj_{L-1}} = 0$  without loss of generality. Now, we use the following change of variable