Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a tensorial representation of data, and linear transformation and split into multiple spectral subsets. An independent neural network is trained on each of these subsets at training and combined through an ensembling approach. This reduces the computational complexity due to smaller data size per network and the parallel training of each subnetwork.
strengths_and_weaknesses: 
**Strengths**

* Able to train an ensemble of subnetworks on a subset of data in a highly parallel fashion is an efficient way to learn generalizable and efficient neural network model.
* This can lead to a paradigm shift in the way models are learned and can enable the scaling of approach to larger data and network sizes

**Weaknesses**
The (novel) contribution of the paper and contrast with literature is not clear. The tensor layer idea seems to be already proposed in the literature. If the parallel implementation and ensembling the network outputs is the contribution, it should be made clear
questions: * In related works, it would be helpful if the proposed approach is contrasted against other ensembling techniques that learn sparse subnetworks [1, 2]. Also, discussion on other data subsetting approaches []  will provide a good context.

* For the fully connected spectral tensor layer, is it possible to set Q < 3n? if so, has the sensitivity to this been evaluated? Similarly, is the effect of the choice of B for the convolutional spectral tensor layer evaluated?

* It is mentioned that the performance metrics of Compression ratio, Parallel Speedup, Convergence, and Accuracy are considered, but these metrics are not explicitly shown in table 3 and 4
* How are the neural network hyperparmeters chosen for MNIST and CIFAR-10 FC-tensor method?
* The performance of tNN and FC-tensor seems to be poor in CIFAR-10 compared to a vanilla FC, why is this?
* Why not use a convolutional spectral tensor layer for CIFAR10.
* Is the O(Q) and O(B) speedup seen during training? 
* A discussion on the limitations of the approach will be helpful. Can this be applied for residual networks?
* Minor typos in multiple places 
   * Strucred linear layers (pg 1)
   * CIAFR-10 used multiple times (pg 8)


limitations: the limitations and potential negative societal impact of this work is not provided
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors proposed a spectral tensor decomposition of a DNN into independent, parallel and computationally cheaper subnets that process corresponding spectrally decomposed data, resulting in speedup in training; ensembling of these subnets yielded reasonable generalization performance. 

strengths_and_weaknesses: [+] Novel idea

[-] Inadequate supporting experiments

questions: - Though a recurrent spectral tensor layer was derived, it was not demonstrated with experiments.  
- The example DNNs experimented are not representative or practically relevant. 
- Do spectral tensor transformers exist?  
- It seems decomposition of data with orthogonal bases is not unique.  Nor is the number of components and the grouping thereof.  How does one choose optimal or reasonable decomposition?  
- With a certain data decomposition, task-relevant information in different components naturally vary.  Corresponding subnet experts might need to have different capacities to realize maximal efficiency?  Or how does one equalize component importance, or do spectral pruning?
- Fig. 2, why was the FC training numerically unstable, with a non-monotonic training loss curve? 


limitations: See above.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces a parallelization of DNN from a tensor perspective. The data is split into their spectral subsets and is passed into parallel subnetworks and finally, the results are based on an ensemble of the subnetworks. Specifically, the author provides detailed spectral counterparts for fully connected layer, convolution layer and recurrent layers. The tensorized structure naturally enables network compression at the same time.
strengths_and_weaknesses: The overall idea is interesting and novel to some extent. By separating data into spectral subsets, the compression limit is extended as well as the level of parallelization. The experimental supports the claim that applying the technic speeds up the training and the inference.


The overall structure is well-organized. However, there may be space for improvement in conveying the motivation, ideas, and justifications.

In the introduction part, the author mentions that the proposed method has no communication cost. It may be better if the author could support such claims by comparing them with other parallelization ideas which have such costs. 

For the main idea, the author mentions the independence of each subset and the orthogonality among each subset. However, how to obtain such subsets and a principal way to select the desired dimension (Q) is not very clear and well-justified. 

My main concern with the work, besides the justification, is how significant and general it is. 

In the experiment session, the author compares performance and runtime on FC, tNN, AlexNet and CycleMLP, but no state-of-the-art architectures on the benchmarks. 

Also there is a lack of runtime comparison on other parallel DNN baselines. 

As for the recurrent settings, the attention mechanism also achieves a certain level of parallelization, which may be worth comparing as well.
questions: 1. how to make sure that the divided subsets are independent and orthogonal to each other.

2. What is the performance (accuracy, runtime, compression) compared to the sota architectures in the benchmark in the experiment session.

3. Is there any way to demonstrate or analyze the generalization ability of proposed structures compared to original structures?
limitations: Limitations not discussed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper introduces a method to train DNN models using parallel ensemble learning. The key idea is to build networks with parallel branches and update each branch independently using split, independent data sets. The evaluation shows that such a method can lead to reduced computation and model size on MNIST, CIFAR-10, and ImageNet. 
strengths_and_weaknesses: Strengths:

- Interesting idea to combine parallel ensemble learning and compression. 
- The paper obtains preliminary promising results on training speedups and model sizes. 


Weaknesses:

- The proposed method in this paper is highly related to parallel training and compression but appears to miss a lot of related work in both areas. On the training side, there are many existing works on reducing the memory consumption of DNN training, such as [1-3]. On the compression side, the proposed method looks closely related to mixture-of-experts (MoE) where a model has multiple parallel expert branches and each branch is sparsely activated during training, but it misses citing any MoE related papers such as MoE[4], Gshard[5], SwitchTransformer[6]. Overall, parallel ensemble learning is an interesting idea and could lead to something quite impactful. However, given its current status of not positioning it well with many existing works on parallel training and sparse training, it is hard to evaluate the significance and novelty of the proposed method. 

There are some mismatched statements. For example, in line 27, the paper says modern DNNs have billions of parameters, but then the examples given are AlexNet, which has only 60 million parameters and is one the earliest network far from representing billion-scale models. 

[1] Rajbandari et al. "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models",  https://arxiv.org/pdf/1910.02054.pdf
[2] Shoeybi et al. "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism", https://arxiv.org/abs/1909.08053
[3] Huang et al. "GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism", https://arxiv.org/abs/1811.06965
[4] Shazeer et al. "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer"
[5] Lepikhin et al. "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding"
[6] Fedus et al. "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity"

questions: Please describe the relationship between the proposed method with respect to existing parallelism strategies for model training and MoE models.

Why choose AlexNet for the evaluation? It would strengthen the work a lot if it is performed on more standard MLPerf benchmarks, such as ResNet or BERT models. 
limitations: No, there is almost no discussion on the limitations of the proposed method. It would be better if the authors could discuss the rationale behind the chosen workloads and also the generalizability of the proposed method.
review_title: null

Meta-review: metareview: Reviewers generally agreed that the parallelization idea described in the manuscript is interesting, but for the most part felt that it is not well positioned within existing literature (making it difficult to gauge the exact novelty), and that empirical results are not sufficiently convincing.  I encourage the authors to address these two critiques and submit their work to a future venue.
confidence: Certain
Decision: Reject