Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces ROAST (Random Operation Access Specific Tile) hashing, a model-agnostic, hardware-aware model compression framework. ROAST essentially provides a global parameter sharing method to give arbitrary control to the user over the memory footprint of model during both training and inference. Evaluation with both BERT and ResNet-9 demonstrates the feasibility of 100x memory footprint reduction without accuracy degradation.

strength_and_weaknesses: **Strengths**

* 100$\times$ reduction of memory usage with no accuracy drop is impressive.

* ROAST addresses the memory usage in both training and serving.

* It is interesting to see only three ROAST operations are sufficient for running DL models.

**Weaknesses**

* ROAST has been evaluated on relatively simple tasks with a small number of classes and it is not clear how it is applicable to larger, more complex tasks without causing accuracy drop.

* The baseline used for evaluation is HashedNet, which seems somewhat outdated.

* In ROAST-MM there is a model-specific parameter $\lambda$, which is determined by "some constant $C$", and it is not clear how sensitive the performance is to the setting of this parameter.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written for the most part. This work builds on the idea of ROBE, but expands its scope to a generalized embedding lookup operation and non-embedding operations.
summary_of_the_review: As the NN scales, there are growing concerns for efficient resource usage for both training and inference--in particular for memory capacity and bandwidth. That said, this paper tackles a timely problem to mitigate this memory bottelenck. On the positive side, the proposed hardware-friendly paramter sharing scheme looks promising and presents impressive results for both BERT-2-2 and ResNet-9. Also, the authors address the memory capacity problem not only for inference (which most existing works tackle) but also for training.

However, I still have several concerns about this work, at least in its current form, as follows:

* ROAST is evaluated on relatively simple classification tasks with a small number of classes. It would have made this work much stronger if the authors evaluate more complex tasks such as ImageNet, SQuAD, and so on.  At the end of the day, the proposed framework would be most useful for large models that can handle complex tasks, but evaluation is currently falling short. I am wondering how ROAST would perform on those complex tasks.

* The baseline used for evaluation is HashedNet, which was introduced in 2015 and hence seems outdated. Is there any stronger, more SOTA baselines to compare against?  How would ROAST compare against them?

* ROAST-MM introduces a scaling factor $\lambda$, which is dependent upon "some constant $C$". It is not clear to me how to set this parameter. Also, how sensitive is the performance of ROAST to the setting of this parameter? How much effort is needed to set this parameter properly? What factor does affect the setting of this parameter? Model? Hardware configuration (like cache size)? Or, both?  Please elaborate.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Weight sharing is one of the effective way to compress the model. This paper propose a new hash method to lower the memory usage for neural network models. Compare with the previous hash method, which use the local memory sharing, this paper propose a new idea call global memory sharing, which is aimed to lower the memory usage. 




strength_and_weaknesses: Strength: Overall, this paper is well organised. Experiment results has well proved the efficiency of the method

Weakness: 1.	Weight sharing can lower the memory usage but can not reduce the number of operations.  As authors claimed, the RAM access is around 100X slower than the computation. This is true. However, from the hardware perspective, a efficient way to lower the memory access delay is cache. The real memory access delay can be much shorter, and can be almost the same as the computation time. Besides, as the computation of neural network is regular, we can use pipelined design to further eliminate the negative impact of memory access. Besides, with the cache design, we can substantially reduce the energy consumption of memory access. I understand this is the common problem of memory sharing, not the problem of this paper only. However, it would be better to do a more comprehensive comparison with other model compression methods not only on model size but also on latency and energy consumption. 

2.	The comparison is not enough. In terms of weight sharing, please do a more comprehensive comparison with other state-of-the-art weight sharing methods. 

3.	Does table 5 compared under the same accuracy? 
clarity,_quality,_novelty_and_reproducibility: The quality of the paper is good. The idea is clearly presented.
summary_of_the_review: Overall, the paper's idea is good in the area of weight sharing. The problem of this paper is the lack of a comprehensive comparison with state-of-the-art. At least it should be compared with more weight sharing methods. 

When compared with model compression method like pruning, the author only list the model size. It is better to also compare the energy consumption and latency.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
The paper presents ROAST, a hashing-based technique for reducing the memory usage of ML models for both training and inference. ROAST improves on the previously existing hashing techniques for this purpose (ROBE/HashedNet) by using block-based (cache/HW – friendly) memory accesses. It also proposes the policy of globally sharing weights (across layers) as opposed to prior art (HashedNet) which shared weights per layer only. This in turn allows ROAST to achieve much higher compression ratios, which translate to much larger performance gains with much lower memory footprint over the state-of-the-art techniques. 

The authors provide theoretical and empirical proof showing that their global memory sharing approach is superior to local memory sharing used in prior art. Lastly, authors demonstrate strong accuracy results using ROAST for a range of Text classification and Image classification benchmarks. 

strength_and_weaknesses: Section 2 and 3 provide a good overview of the targeted problem space and the immediate baselines (ROBE/HashedNet) which are closest to the proposed ideas. The diagrams provided in the paper are helpful in explaining the proposed ideas and authors also present detailed proofs that help justify the superiority of GMS approach over LMS. Authors show large speed up compared to the previous hashing approach, HashedNet. 

In Section 6, authors explain the experimental setup designed to verify the efficacy of ROAST. While the authors explain that they perform some hyperparameter tuning to achieve a given target accuracy it fails to communicate how using GMS approach in ROAST impacts the trainability or ease of training of their models. E.g., Adding the epochs of training incurred to achieve the target accuracy for the different baselines in Table 3 and 4, might help explain that. The intuition is this, since the weights are shared across all layers the cumulative gradient update application might make it harder to get the same level of accuracy with fewer unique weights (compared to LMS).

In Table 4, authors demonstrate that ROAST can be combined with other model compression techniques e.g., PRUNING to achieve further model compression. While this is lucrative, structured pruning that takes advantage of the block-wise nature of ROAST memory access could achieve even better accuracy with low training effort. Consider trying that as well. 

Table 5 shows the inference runtime for a range of memory size and model size values on a GPU with 48MB cache. While the ROAST inference time dominates over HashedNet baseline, especially for large model sizes (high compression ratio). The comparison with PyTorch based matrix multiplication (MM) is also interesting. While PyTorch with its efficient matrix multiply implementation achieves fast inference time for lower model sizes, as the model size grows the benefit of PyTorch over ROAST reduces. Hinting potentially that ROAST based memory optimization could further help improve PyTorch MM implementation for hashed DNNs. 
The results in Figure 3b are somewhat contradictory to that result that GMS is superior to LMS for a range of models. Considering that the Local ROAST approaches (10x and 100x) end on a slightly higher accuracy number than corresponding GMS approaches. Even seeing accuracy values that match closely for GMS and LMS approaches is surprising. Kindly explain why the yelp-polarity model defies the expected trend as seen in Figure 3a. 

I appreciate the authors clearly stating the limitations of the proposed work (focus on reducing memory access during training/inference only, not compute cost)

clarity,_quality,_novelty_and_reproducibility: Questions related to quality of results are discussed in strengths and weakness section.
summary_of_the_review: The results show in Figure 3b) violate the statement proved showing GMS should achieve higher performance than LMS. Without a good justification for this miss, it is hard to confirm that the GMS technique has been effectively scaled to all DNN layers. Further, it is important to understand if the GMS approach brings with it any addition to training complexity (e.g., increased hyperparameter tuning). If the authors can add data for these requests, it will help consolidate the presented results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model compression approach: Random Operation Access Specific Tile (ROAST) hashing. The authors consider three operations for block-based hashing to reduce memory usage, and they also introduce a global memory-sharing method to improve model accuracy. Experiments on BERT and ResNet show that the proposed ROAST exceeds HashedNet and achieves high compression when training models.
strength_and_weaknesses: ## Strength
1. The proposed ROAST achieves similar quality in almost 100x less space for compression model training. ROAST performs well in many research areas including text classification and image classification. The experimental results show that the proposed global memory sharing method performs better than previous local memory-sharing. 

## Weakness
1. Lack of comparison of time costs between GMS and LMS. Will this global memory share cause access conflict?

2. For Local vs. Global memory sharing in Figure 3(b), the GMS shows more significant accuracy degradation than LMS. How do you know this is not relevant to the GMS method but to other factors? 

3. Lack of experiments on different models and datasets. Many text-classification tasks are introduced in the introduction but only experiment on BERT-2-2 and BERT-base. For the image classification task, only ResNet-9 on CIFAR-10 is used.

4. Need more details about the implementation of ROAST operations. It is hard to follow.

clarity,_quality,_novelty_and_reproducibility: Please see the questions above. 
summary_of_the_review: While the paper proposes an interesting idea, the experiments are insufficient, and some details of the proposed technique are missing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper proposes a model compression approach: Random Operation Access Specific Tile (ROAST) hashing. The authors consider three operations for block-based hashing to reduce memory usage, and they also introduce a global memory-sharing method to improve model accuracy. Experiments on BERT and ResNet show that the proposed ROAST exceeds HashedNet and achieves high compression when training models.

Strength: Overall, this paper is well organised. Experiment results are promising, e.g., (100 reduction of memory usage).

Weakness: The scope of this paper is still limited because the metrics like energy consumption were not included. It would be better to do a more comprehensive comparison with other model compression methods not only on model size but also on latency and energy consumption. The benchmark CV datasets is small scale (i.e., CIFAR10) why not try large-scale CV datasets? No justifications are provided. 

All the reviews have the consensus on the down side for this paper. 
justification_for_why_not_higher_score: More experiments (latency and energy consumption, larger CV datasets) are needed to support the claims. 
justification_for_why_not_lower_score: NA
Decision: Reject