Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This papers proposes a way to speedup the training of GBDTs via quantization technique. They adopt the idea of quantized gradients that is widely used in neural net training/compression literature and show that it is possible to reduce the precision as low as 2-3 bits and yet to achieve similar performance in terms of accuracy. The overall speedup in training time can be up to 2x.
strengths_and_weaknesses: Strengths:
- Practical and simple approach to speedup GBDT training for both CPU and GPU;
- The paper is clearly written with nice introduction to GBDT training preliminaries and techniques which is especially useful for non-experts
- Section 6 describes system implementation details which is quite interesting. I especially liked the idea of packing the gradients and hessian.
- It is quite surprising to see that lowering the precision that much (up to 2-3 bit) would preserve the accuracy. It definitely has research/practical value and raises an interesting question: are GBDTs overparameterized? It is not a secret that the ensemble size in production environment can be huge. And if that is the case, then should we consider compressing them?

Weaknesses:
- Authors provide theoretical analysis of the method (i.e., error bound after quantization) which is great. However, there are some limitations: 1) the main theorem is only valid decision stumps; 2) the bound is presented for EACH GRADIENT. How about their summations or a bound for histogram counts? Moreover, the error can be proportional to max|$g_i$| (plus square of it) which I believe is pretty high and not sure if it is useful result.
- Novelty. The training of GBDTs is done in traditional way: gradient+hessian, feature histograms, etc. So, there is no contribution from that angle. The application of quantization here is also straightforward and done via binning and rounding. However, it seems there is no previous work which combines them, whereas this paper shows its huge benefits.


----- After Rebuttal ------------

My main concerns regarding this paper were addressed during rebuttal and I'm willing to increase my score to 7. Overall, I think that this paper is a nice contribution with both theoretical and practical results. Especially, experimental results suggest strong implications: one can significantly quantize (compress) gradients and still learn pretty accurate boosted trees. Novelty seems to be fine: although quantization is well known technique in NN literature, this paper seems to be the first who proposes applying it to trees and did a nice job on that.
questions: .
limitations: .
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work proposes a low-precision training algorithm for GBDT based on gradient quantization and theoretical analyze the necessary precisions of gradients without hurting performance can be quite low. The authors also conduct extensive experiments on both CPUs and GPUs and the results show great improvements.


strengths_and_weaknesses: Pros:

1. The article is well written, quite clear.

Cons:
1. The advantages and limitations of the approach could be better underlined.
2. The work is limited in its novelty.
questions: 1. Whether the quantized training of GBDT would have a negative influence on the inference performance ?  
limitations: 1. There are some other limitations that I think should be more discussed
2. The work is limited in its novelty.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a quantised version of Gradient Boosted Decision Trees. The paper proposes to quantise the high-precision gradients in a rather simple way. The paper considers the problem how many bits are really needed for gradients in GBDT to achieve a reasonable performance. It is shown that with the low-precision gradients most arithmetic operations may be replaced by integer operations. 
strengths_and_weaknesses: Strengths. The paper is clearly written. The results both theoretical and numerical are convincing. 

Weaknesses. The idea to perform training with the quantised gradients is not new. However, the papers introducing the quantised learning which appeared 5-6 years ago (Binarised neural networks, results of M. Courbariaux and Y. Bengio) are well cited. 

questions: The last paragraph on page 3 is not extremely clearly written, e.g., what is meant under "And construct histograms with the bins"? 

Maybe I missed it : the results reported in Tables 2 and 3 comparing various boosting algorithms: are the number of estimators and the depth of trees the same in all these methods? How many estimators (trees) and what is the depth?
limitations: It would be interesting to discuss the cases (if they exist) where quantised gradients lead to much worse performance than continuous ones.
review_title: null

Meta-review: metareview: The reviewers conclude on an interesting paper with broad messaging that does make sense -- I do subscribe to it as well. I recommend it for acceptance, noting that interactions with reviewers were the occasion to provide additional remarks that have to be used to craft the camera ready, in particular for the remarks made to PPQi (mostly the technical part of this discussion).
confidence: Less certain
Decision: Accept