Title: Parallel Stochastic Gradient Descent with Sound Combiners
Authors: Saeed Maleki
Madanlal Musuvathi
Todd Mytkowicz
Yufei Ding
Venue: ICLR 2017

Abstract: Stochastic gradient descent (SGD) is a well-known method for regression and classification tasks. However, it is an inherently sequential algorithm — at each step, the processing of the current example depends on the parameters learned from the previous examples. Prior approaches to parallelizing SGD, such as Hogwild! and AllReduce, do not honor these dependences across threads and thus can potentially suffer poor convergence rates and/or poor scalability. This paper proposes SymSGD, a parallel SGD algorithm that retains the sequential semantics of SGD in expectation. Each thread in this approach learns a local model and a probabilistic model combiner that allows the local models to be combined to produce the same result as what a sequential SGD would have produced, in expectation. This SymSGD approach is applicable to any linear learner whose update rule is linear. This paper evaluates SymSGD’s accuracy and performance on 9 datasets on a shared-memory machine shows up-to 13× speedup over our heavily optimized sequential baseline on 16 cores.

TL;DR: This paper proposes SymSGD, a parallel SGD algorithm that retains the sequential semantics of SGD in expectation.

Review discussion:
Specifically, it is unclear from the paper whether does the gradient calculation utilizes the sparsity of the dataset.
It would be helpful to give a detailed description on how gradient is calculated, and time complexity analysis, given  N X M a sparse input data, with only Z entries
We do utilize the sparsity of the dataset for gradient computation in the case of sparse datasets.
However, our algorithm works for both sparse and dense datasets – the correctness and speedup extends in both cases. As demonstrated in our experiments, RCV1 for example is 99.84% sparse and Epsilon is 0% sparse. Our implementation ensures that when operating on a sparse dataset, the corresponding operations are proportional to the number of non-zero entries.
Specifically, the gradient as equation (1) shows is (X_r \cdot w_{i-1} – y_r)X_r^T where X_r is a sparse or a dense instance. Let’s assume that X_r has O(z) number of non-zeros.
Therefore, gradient computation involves a dot product computation of X_r \cdot w_{i-1} which costs O(z) in time and a scalar-vector multiplication which also costs O(z) in time. This means that an iteration of our baseline algorithm takes O(z) amount of time. For the same instance, SymSGD takes O(z x k) amount of time where k is the size of projected space.
We have uploaded a revision of the paper with this clarification. The third paragraph of Section 3 discusses the complexity of sequential SGD and the last two paragraphs of Section 3.2 discuss the complexity of SymSGD.
Perhaps I did not read careful enough, but how do you choose the size of k? Since per instance time of SymSGD is k times larger than the baseline, it seems k need to be small in order to have a speed gain over the baseline
You are correct, overhead of SymSGD per thread is proportional to k.
However, what is surprising is that k across all of our benchmarks can be so small without loss of accuracy (k is between 7 to 15). For such a small k, we are able to hide most of the overhead by utilizing SIMD hardware within a thread. As it can be seen from Figure 2, the slowdown of SymSGD with one thread over the baseline which runs sequential SGD without computing the model combiner is less than 50%. Since SymSGD enables a parallel way to combine learned models, it enables multicore parallelism speed up.
Like other algorithmic parameters, we did a parameter sweep to find the best performing k.
We made the changes in the last two paragraphs of Section 3.2 of the revision to discuss this.
"In parameter-server Li et al. (2014a), each thread (or machine) periodically sends its model deltas to a server that applies them to a global model. In ALLREDUCE Agarwal et al. (2014), threads periodically reach a barrier where they compute a weighted-average of the local models. Obviously, these approaches can produce a model that is potentially different from what a sequential SGD would have produced on these examples."
The convergence of asynchronous SGD has been theoretically proven. Why do you think it is "potentially different from" the sequential algorithm?
We agree that as written our paper seems to claim that other algorithms do not converge. Our intention was to claim that these algorithms can potentially learn a different model than SGD after processing a certain number of instances, even if these algorithms eventually converge. Our algorithm produces the same answer as sequential SGD in expectation. We modified the sentence in our paper with this clarification. The uploaded revision includes this change. Our experiments (Fig 2) indicates that while HogWild learns models with equally good error rates after processing the same number of examples, AllReduce clearly falls behind.
This paper describes a correction technique to combine updates from multiple SGD to make it statistically equivalent to sequential technique.
Comments
1) The proposed method is novel and interesting to allow update to be corrected even when the update is delayed.
2) The proposed theory can only be applied to square loss setting (with linear update rule), making it somewhat limited. This paper would be much more interesting to ICLR community, if the technique is applicable to general objective function and settings of deep neural networks.
3) The resulting technique requires book-keeping of a dimensional reduced combiner matrix, which causes more computation in terms of complexity. The authors argue that the overhead can be canceled with SIMD support for symbolic update.
However, the normal update of SGD might also benefit from SIMD, especially when the dataset is dense.
Overall, even though the practical value of this work is limited by 2) and 3), the technique(specifically the correction rule) proposed in the paper could be of interest to people scaling up learning. I would encourage the author to extend the method to the cases of non-linear objective function which could make it more interesting to the ICLR community
We would like to thank you for your review and address comments 2 and 3 in here.
Regarding comment 2: As stated in the paper, our technique is limited to learners which have a linear update rule for the weight vector. This includes linear or polynomial regression with squared loss function and L2 regularization. The only other approach to parallelize on a shared-memory system for such a problem is Hogwild which as our results show, does not scale across sockets in a multiple socket system. We are currently working on extending our approach to non-linear objective functions.
Regarding comment 3: Utilizing SIMD requires regular accesses. Sequential SGD can benefit from SIMD only when the dataset is completely dense – every instance has values for every feature. In our collection of 9 datasets, epsilon is the only one with such 100% density – other datasets such as MNIST or ALOI are relatively dense but nowhere near 100% density required to use SIMD well.
By linear regression I realy mean quadratic loss function, I have updated the review respectively.
Overall, the idea in this paper is interesting and the paper is well-written and well-motivated.
However, I think it is not ready to publish in ICLR for the following reasons:
- This paper is not related to representation learning. It may be more suitable for a general machine learning or data mining conference.
- The proposed approach can only work for a small class of models and cannot apply to popular formulations,  such as SVM, logistic regression, and neural network. It is unclear why we want to use SGD for this specific type of formulations. For model like linear regression, the authors should compare their methods with linear programming approaches.
Also, it is unclear why we need to develope parallel algorithm for linear regressio problems as they are relatively easy to solve unless the data are big (see next comment).
- The dataset used in the paper are relatively small and can be only used for proving the concept. Most datasets considered in the paper can be solved in a few second using a single core CPU. Hogwild! is suitable for sparse dataset because of its asynchronized nature. On data that are very sparse, the proposed approach is only slightly better or is worse than Hogwild. For dense dataset, it is unclear why we need to use SYMSGD instead of simply parallelizing the gradient computation using GPUs. Put them together, the experiment results are not convincing.
Regarding other popular formulations:
We compared the maximum accuracy that Vowpal Wabbit can achieve using logistic and linear regression and table 1, columns 7 and 8 compare the results. As it can be seen, among the datasets we examined, the difference is very minimal and in some cases linear regression achieves even better accuracy.
Therefore, we believe studying performance of linear regression is valuable.
Regarding linear programming approaches:
One of the best existing least square linear solver is Intel MKL implementation of LAPACK which uses Singular Value Decomposition method (gelsd). It currently only supports completely dense datasets – those in which every instance has a value for every feature. For our only completely dense dataset, epsilon, we evaluated the performance of MKL in comparison to our single-threaded baseline:
a.	MKL using 16 cores takes around 20.64 seconds to converge and gives an accuracy of 89.62%
b.	Our baseline using 1 core takes around 17.66 seconds to converge to an accuracy of 89.71%
Clearly, even our SGD baseline approach is faster than MKL with 16 cores.
Moreover, SymSGD using 16 cores is 9x faster than MKL using 16 cores.
For datasets that are not completely dense, (including “dense” datasets which still have some features missing per instance,) we have to manually densify the dataset which clearly increases the overhead of the whole computation and requires excessive amount of memory. An alternative would be to compute X^T . X which is an fxf sparse matrix and then use MKL PARDISO parallel sparse linear solver to compute the exact solution. There are two issues with this approach: (1) X^T . X needs to be full-rank since MKL can only find the answer when it is unique. It is very unlikely that X^T . X is full-rank since the features of a dataset can be linearly dependent.
Also, even if X^T . X is full-rank, the unique answer can be overfitting and undesired. (2) Computing X^T . X requires O(f^2 . z) time where z is the average number of non-zeros across columns of X. This is also infeasible for datasets with large number of features. (For example, url has 3.2 million features and the corresponding fxf matrix requires an excessive amount of time.)
Regarding the size of our datasets and their training time:
Not all of our datasets are small in size.
For example, the training datasets for mnist8m, epsilon and url are 19GB, 11.8 GB and 1.5 GB, respectively.
However, the size of a dataset by itself is not the only indication of the training time and number of classes has an important factor in the time since we need a model for each class.
For example, our baseline takes 371 seconds on mnist8m with 10 classes, 124 seconds on sector with 105 classes and 59 seconds on url with 1 class until they converge to the maximum accuracy.
Therefore, the training which takes minutes to end can take seconds to converge using our SymSGD parallel algorithm.
Regarding Hogwild being faster:
Among our 9 datasets which is combination of datasets with different sparsity, Hogwild is faster in only 2.
But the important conclusion from our results is that Hogwild does not scale beyond one socket of processor while SymSGD keeps scaling (in most cases, by going from 8 to 16 threads, Hogwild’s performance drops).
Regarding using GPUs for dense datasets: SymSGD’s parallelism is across iterations of the SGD loop and it is orthogonal to any other available source of parallelism within each iteration of SGD.
For example, we may use multiple GPUs where each one computes a local model and a model combiner. This way we have the massive parallelism of GPU within an instance while across multiple GPUs we have the parallelism of SymSGD.
However, note that there is not that much parallelism available even in a dense dataset.
For example, mnist8m which is one of the biggest datasets in our study has on average ~200 non-zeros. Parallelism across 200 floating point operations can be tricky.
This paper propose a parallel mechanism for stochastic gradient descent method (SGD) in case of gradient can be computed via linear operations (including least square linear regression and polynomial regression problems). The motivation is to recover the same effect compared with sequential SGD, by using a proposed sound combiner. To make such combiner more efficient, the authors also use a randomized projection matrix to do dimension reduction. Experiments shows the proposed method has better speedup than previous methods like Hogwild! and Allreduce.
I feel that there might be some fundamental misunderstanding on SGD.
''The combiner matrixM  generate above can be quite large and expensive to compute. The sequential SGD algorithm maintains and updates the weight vector w , and thus requires O(f)  space and time, where f  is the number of features.
In contrast,M  is a f f  matrix and consequently, the space and time complexity of parallel SGD is O(f^2) .
In practice, this would mean that we would need O(f) processors to see constant speedups, an infeasible proposition particularly for datasets that can have
thousands if not millions of features."
I do not think one needs O(f^2) space and complexity for updating M_i * v, where v is an f-dimensional vector. Note that M_i is a low rank matrix in the form of (I - a_i a_i'). The complexity and space can be reduced to O(f) if compute it by O(v - a_i (a_i' v)) equivalently. If M_i is defined in the form of the product of n number of rank 1 matrices. The complexity and space complexity is O(fn). In the context of this paper, n should be much smaller than f. I seriously doubt that all author's assumptions, experiments, and strategies in this paper are based on this incorrect assumption on space and complexity of SGD.
Why one can have speedup is unclear for me. It is unclear what computations are in parallel and why this sequential algorithms can bring speedup if M_i*v is computed in the most efficient way.
I suggest authors to make the following changes to make this paper more clear and theoretically solid
- provide computational complexity per step of the proposed algorithm
- convergence rate analysis (convergence analysis is not enough): we would like to see how the dimension reduction can affect the complexity.
Regarding the compact representation of M:
You are right that M has an O(n . f) representation, which is compact for datasets with n << f. Unfortunately, such a representation defeats the purpose of a parallel algorithm because computing M.w from this representation performs almost the same sequential computation as the sequential SGD (a dot product of an instance with a weight vector followed by a vector addition). By repeating the (sequential) work in the reducers we will not get parallel speedups. For speedups, we want the work in the reduction to be independent of n (we shouldn’t have to revisit the training data) and perform only a small fraction of the cost of the learners (otherwise the Amdahl's law will limit our speedup).
Also, from Table 1, n > f for many of our datasets and n > f/10 for all our datasets.
But your observation and the discussion above is relevant for other datasets and we will add them in the paper.
We do utilize the "low-rank" representation of M in other places in the paper.
For instance, projecting N = M-I instead of M relies on this property. We also use this representation to efficiently compute M_A = M*A, the projected version of M.
Regarding the complexity of SymSGD:
This is a great point and we will add the discussion in the paper (one other reviewer also brought this up and we have a revision that addressed his/her concern).
Let n be the number of instances in the training dataset, f be the total number of features, f’ be the average number of non-zeros per instance, c be the number of classes, k be the dimension of the projected space, and p be the number of processors.
Therefore, the local learner complexity for all processors per pass is O(n . f’ . (k+c)) in time and O(p . f . (k+c)) in space. The reduction complexity for all processors is O(p . f . k . c) in time and O(p . f . (k+c)) in space.
Regarding convergence rate analysis of SymSGD:
By producing the same result as a sequential SGD in expectation, SymSGD enjoys the same theoretical convergence rates as SGD. We will be happy to formalize this in a theorem in the paper. Our empirical results confirm this fact.

Meta-review:
role: PC
data: comment: The reviewers largely agree that this paper is well written and presents an interesting, novel approach to parallelizing Stochastic Gradient Descent. However, the current formulation is restricted to linear regression models and requires sketching techniques to handle large number of features, although it is striking that very aggressive sketching (k~10) still works well. In this setting though, there are specialized randomized solvers such as Blendenpick (Avron et al) which sketch the data upfront to construct a high quality pre-conditioner for use with iterative methods. 
 The authors are encouraged to either compare with state of the art parallel randomized least squares solvers developed in the numerical linear algebra community (see papers by Michael Mahoney, Petros Drineas and others), or broaden the scope of the proposed methods to include models of current interest (e.g., DNNs). The latter would of course be of specific interest to the ICLR community.

Decision: Reject