Addition is almost all you need: Compressing large language models with double binary factorization
Authors: Vladimír Boža
Vladimír Macko
Venue: Accepted by TMLR ()

Abstract: Binary quantization approaches, which replace weight matrices with binary matrices and substitute costly multiplications with cheaper additions, 
offer a computationally efficient approach to address the increasing computational and storage requirements of Large Language Models (LLMs).
However, the severe quantization constraint ($\pm1$) can lead to significant accuracy degradation.
In this paper, we propose Double Binary Factorization (DBF), a novel method that factorizes dense weight matrices into products of two binary (sign) matrices, each accompanied by scaling vectors. 
DBF preserves the efficiency advantages of binary representations while achieving compression rates that are competitive with or superior to state-of-the-art methods.
Specifically, in a 1-bit per weight range, DBF is better than existing binarization approaches. In a 2-bit per weight range, DBF is competitive with the best quantization methods like QuIP\# and QTIP. 
Unlike most existing compression techniques, which offer limited compression level choices, DBF allows fine-grained control over compression ratios by adjusting the factorization's intermediate dimension.
Based on this advantage, we further introduce an algorithm for estimating non-uniform layer-wise compression ratios for DBF, based on previously developed channel pruning criteria.
The code is available at: \url{https://github.com/usamec/double_binary}

Published in Transactions on Machine Learning Research (03/2026)
Addition is almost all you need: Compressing large language models with double binary factorization
Vladimír Boža
boza @fmph.uniba. uniba. sk
Faculty of Mathematics, Physics and Informatics Comenius University, Bratislava
Vladimír Macko
vladimir. macko Of finphh uniba.
Pacully of Mathematics, Physics and Informatics Comenius University, Bratislova
Reviewed on OpenReview: https: // openrevee. net/ met/ forum? id= kōkUKoewd)
Abstract
Binary quantization approaches, which replace weight matrices with binary matrices and substitute costly multiplications with cheaper additions, offer a computationally efficient ap- proach to address the increasing computational and storage requirements of Large Language Models (LLMs). However, the severe quantization constraint (+1) can lead sigmificant accuracy degradation. In this paper, propose Double Binary Faclorization (DBF), novel method that factorizes dense weight matrices into products of two binary (sign) ma- trices, each accompanied by scaling vectors. DBF preserves the efficiency advantages of binary representations while achieving compression rates that are competitive with or su- perior to state-of-the-art methods. Specifically, in a 1-bit per weight. range, DBF is better than existing binarization approaches. In a 2-bit per weight range, DBF is competitive with the best quantization methods like QulP# and QTIP. Unlike most existing compression techniques, which offer limited compression level choices, DBF allows fine-grained control compression ratios by adjusting the factorization's 's intermediate dimension. Based on this advantage, further introduce algorithm for estimating non-uniform layer-wise compression ratios for DBF, based on previonsly developed channel pruning criteria.
The code available at: https://gituu..com/usaaaea binara
1 Introduction
Large language models (LLMs) have achieved unprecedented success in various natural language processing and reasoning tasks. However, the increasing scale of these models has led to substantial computational and storage demands, posing significant challenges for deployment. To address these limitations, various compression techniques such as quantization (Frantar et al. 2022 Malinovskii et al. 2024 Tseng et al.
2024a b), and pruning (Frantar & Alistarh 2023 Boža 2024 have reemerged, aiming to reduce model size and inference latency without significant loss in performance. Moreover, methods like BitNet (Wang ett al. 2023) and OneBit {Xu et al. 2024) restrict weight matrices to binary values and replace energy-costly multiplication with more energy-efficient addition.
In this paper, we push weight binarization further. We propose to replace each weight matrix with product of two binary sign matrices, each scaled by appropriate vectors. We present a heuristic algorithm for calculating such factorization and show that it results superior compression compared to a single sign matrix and competitive with the state-of-the-art quantization approaches.
Summary of contributions. We propose practical algorithm for factorizing dense weight matrices into product of two binary matrices (with appropriate scaling factors). We apply this factorization to

---

Published in Transactions on Machine Learning Research (03/2026)
DBF
Wikitext PPL
9
AQLM+PV
QTIP
8
X
OneBit
W
QuiP#
6
1.0
1.5
2.0
X
A+1
Avg. bits per weight
b
B+
Figure 1: Left: Schematic drawing of computation in Double binary factorization. X is the layer input FP16 format, W the original weight matrix. A and B consist of +1 clements, a, m, are vectors in FP16 format. We omit matrix transpositions for simplicity. Right: Comparison of our Double binary factorization (DBF) with previously proposed network compression methods Llama2-7B.
LLM compression and achieve results better than single-matrix binarization and competitive with leading quantization methods, Our factorization is practical for deployment current GPUs, achieving 2-3.5x speedups during inference. Furthermore, since multiplication with binary matrices requires only additions, our approach has potential for significant energy savings.
DBF also has multiple notable advantages compared to other weight compression methods. First of all, it can take weight importance into account and gives lower error lo weights with higher importance. Furthermore, DBF can also achieve fine-grained range of compression ratios by varying the size of the middle dimension of the factorization. Typical compression approaches allow only limited choices of compression ratios and are commonly limited to integer number of bits per weight. Finally, we propose iterative algorithm to assign different compression ratios to each layer. We show that we treat the middle dimension of factorization as channels and use previously devised channel pruning criteria to iteratively prune the middle dimension of factorization.
2
Related work
Post- Tost-Training quantization (PTQ). There many methods for post-training quantization of LLMs.
Almost all of them work by solving some variation of the layer-wise compression problem. GPTQ (Frantar
et
al.
2022) improves scalar quantization by minimizing layer-wise errors. QuIP# (Tseng et al., 2024a
uses
incoherence preprocessing and lattice codebooks to further improve quantization accuracy. AQLM
(ggiazarian et al.
2024h uses learned vector codebooks. QTIP (Tseng et al. 2024b) combines incoherence
preprocessing with trellis coding to get even better accuracy than previous methods. However, a key draw- back of these more sophisticated quantization methods is that they require docompressing weights back full precision and performing full-precision multiplications, thus preventing them from utilizing hardware acceleration optimized for lower-precision arithmetic. Additionally, most existing methods support only limited set of compression ratios (c.g., QTIP needs an integer number of bits per weight)), reducing their flexibility.
Models with binary weights. Networks with binary weights (Rastegari et al. 2016) were always attrac- tive, due to the potential of simple computation, where one replaces multiplication with additions. This idea was applied LLMs in BitNet (Wang al. (023),, which showed possibility of training LLMs with binary weights. OneBit (Xu et al. 2024) showed simple way of converting each weight matrix into binary
2

---

Published in Transactions on Machine Learning Research (03/2026)
matrix during post-training quantization. BiLLM (Huang et al. 2024 also binarizes weight matrices, but uses another binary matrix to enhance precision for selected subset of important parameters.
Weight factorization. Rather than quantizing weight matrices, another approach is to approximate them as a product of several smaller matrices. The most common method is low-rank factorization (Li & Shi 2018 Jaderberg et al.
2014), where an n x m matrix is decomposed into the product an n lt and k m matrix, with < min(n, (2, m) However, low-rank factorizations come with severe degradation in accuracy. [Bžaa
& Macko (2025) showed that one can factorize the weight matrix into two sparse matrices and get better results than using just sparse matrix with the same amount of nonzeros. Saha et al. (0224] proposed Caldera, which decomposes cach weight matrix sum of high-precision low-rank factorization and low- precision quantized matrix. SVD-LLM & SVD-LLM v2 (Wang et al.) 2024 2025) improve over naive SVD by incorporating truncation- data whitening to align singular value truncation with compression loss and assigning specific compression ratios based on the varying sensitivity of different layers. [Cherniuk et al.) (2024) factorics convolution layers in vision models into product of multiple quantized matrices. Compared to work, they single scaling factor for the whole tensor and much higher precision (4-bit and higher).
Finetuning of quantized models. State-of-the-art LLM compression methods use fine-tuning during and after quantization. Multiple methods fine-tune only the remaining continuous parameters. AQLM [Egiazarian et al. 2024] only finetunes continuous parameters after the quantization. Qul (Tseng et al, 2024a) and QTIP (Tisng et al. 2024b) use lightweight fine-tuning during the quantization process and then proceed to fine-tune continuous parameters after quantization.
In most cases, the fine-tuning of discrete parameters of quantized models is done via straight-through estima- tion Courbariaux et al. 2014 Jacob et al. 2018). The other option is to use stochastic rounding (Alistarh) et al. 2017p. Malinovskii et al. (2024) proposed PV-tuning, where they showed that it is better to tune discrete parameters with a higher learning rate, but only tune a small subset of discrete parameters in each update step.
3 Double binary factorization
We work in the context of layer-wise compression, where each linear layer in the model is replaced by a compressed version. Our goal is to keep weight matrices in binary format while having regular 16-bit floating-point activations.
OneBit decomposition.
Xu et al. 2024 proposed to approximate weight matrices by:
W W W
where a, b are column vectors, W+1 is a sign matrix with entries from {-1,1} and @ denoles elementwise (Hadamard) product with appropriate broadcasting. During linear layer computation, we calculate:
XW ~ ((X @ 2 (F))
OneBit decomposition is calculated via Sign- Value- Independent Decomposition (SVID), which first sets W41 Sign (W) and ab" is the rank-1 approximation of [W] calculated via SVD or NMF.
Double binary factorization. Inspired by Double Sparse Factorization (Boža & Macko 2025 propose to factorize the weight matrix into product of two binary matrices with appropriate vector scaling in between (see also Fig. 1
W = (a A4 @ m TTTT (B -b))= (a A+1)(m @ B+1 Ob")
where m, b are vectors and A1ı, Bui are sign matrices, which translates into the following computation during network forward pass:
XWT ((((X @ oF ) の m TT4A) o a
3

---

Published in Transactions on Machine Learning Research (03/2026)
Middle dimension size. The original matrix W has shape and respectively. But we can select the middle dimension size k to match the desired compression ratio. Then the vector m will have size k and matrices A=1 and B+1 will have size x k and k x m.
Vectors and b must have sizes n
For example, the imput matrix squared and we target approximately 1-bit compression, we will select k = n/2. If we target ~2-bít. compression, we will select n. In general, we will select k b nm where is the desired average number of bits per original weight.
u+m!
3.1 Practical considerations for double binary factorization
Storage size. By controlling the middle dimension of factorization, we can compress the matrix to any desirable size. This is a huge advantage compared many other methods, which support limited number of sizes (e.g., scalar quantization supports only integer values as number of bits per weight). We also need to store scaling vectors, but their size is negligible (e.g., when compressing a matrix of size 4096 x 4096 to 2 bits/weight, storing scaling vectors costs 0.012 bits per weight).
Inference costs. is true that the total number operations for DBF is higher than for ordinary scalar quantization (except for the case where the number of bits per weight is less than or equal to 1), but in all cases, we replace costly multiplications with much simpler additions. Moreover, LLM inference is often bound by memory transfer, and memory transfer costs for DBF are similar to the costs of any other quantization with an equal number of bits per weight. We measure actual matrix multiplication timings in the experimental section and found that DBF is 2-3.5x faster than dense bascline when using 2 bits per weight. Moreover, Wang et al. (2023) showed that using binary weights can save energy during inference since one needs only additions instead of multiplications. This could lead to even better savings in the future with proper HW support.
Storage of middle activations during finetuning. As with any other weight factorization, one needs to store the middle activations during fino-tuning, which incurs nontrivial memory costs. Similar to Double sparse factorization (Boža & Macko 2025), we found that when using gradient checkpointing, this storage increase is practically negligible.
3.2 Computing Double Binary Factorization
Computing optimal Double binary factorization is probably an NP-hard problem, but we respond to that with 'Here's where the fun begins." (Solo, 1977) and propose heuristic algorithm for computing DBF. We compute Double Binary Factorization very similarly to Double Sparse Factorization (DSF) (Bžaa & Macko, Our main goal to minimize:
2025
min ||W (a o A+1 O mB C 0F
First, we split the middle scaling factor into two (we can easily put it back by multiplying the middle factors together):
min || - @ @ mm2 C Bi @
Denote A a @ A+1 O of and B = m2 O B+1 ② 6T, We run alternating minimization, where first initialize A with random matrix, then fix A and optimize B, then fix B and optimize A, elc.
The main subproblem of the alternating minimization is:
min A
||B - W
8. t. A A - om
We solve this problem using the Alternating direction method of multipliers (ADMM) (Boyd et al. 2011 ADMM is iterative algorithm for solving constrained optimization problems. While ADMM provably
4

---

Published in Transactions on Machine Learning Research (03/2026)
converges only for convex problems, and our constraint is non-convex, there have been numerous successful attempis to use ADMM for non-convex problems.
We use ADMM for solving constrained optimization of the form:
minimize f(x)
subject to x EEC
In this case, the one iteration of ADMM is (p is penalty factor, usually set to in case, and scaled dual variables):
x k+1 arg min f(x) + (p/2)]la 2t u1
k+1 [1 n² u k+1 = w + x .k1
Where Tee is Euclidean projection onto the C. Tn case, want to project onto the set of matrices, which be factorized as @ A. O mf We will use SVID projection from OneBit (Xu et al. 2024). i.e., we will calculate SVID(Z) (2) as follows: First, we set Z+1 = Sign(Z) [), and do rank-1 decomposition of Z into amf- Then SVID(Z) = a @ Z11 @ mf. Since we need to do a lot iterations, we compute the rank-1 decomposition using power iteration.
Then the one ADMM update becomes:
((++1) = (B pl) (B" + - -((())
( = SVIID SVID(W((--)) + U(K)
((++1) = Uk + W(++1) - (1+1))
We also use all heuristic improvements from DSF (warm-starting of inner iterations). We also normalize rows of matrix B; we prefer to use fewer inner updates (ADMM steps) and more outer updates (alternating minimization steps), and we reuse solutions from previous inner iterations.
3.3 Input and output importance scaling
While many compression schemes try to solve the layer-wise pruning problem (min |XW - -XW W.ll3. where X is the calibration input, W the original matrix, and Wa the compressed matrix) (Frantar & Alistarh 2023 Frantar et al. 2022 Tseng et al. 2024a), we opt to use a slightly different approach.
We assign different importance to proserving the rows and columns of the matrix W. We use input activation (similar to Wanda (Sun al. 2023)) the column (input) importance. We the gradient norm (output) importance. Both input activation norms and gradient norms precalculated using small calibration dataset. One can think about scaling by activation norm and gradient norm crude rank-1 approximation to the diagonal Fisher matrix.
We can casily incorporate input and output importance into our factorization algorithm. We first calculate 2 W C i", where o are gradient norms and i are input activation norms. We then factorize W (a'
W
o
A+1 G (B41 DT)) and scale scaling vectors back: a a' /o, b/i
3.4
during and after factorization
Almost all state-of-thar toite-of-thh-att quantization algorithms use some form of fine-tuning. We use a slight variation of the procodure from QulP# [Tsong et al. (0244) and QTIP [Tseng et al.] 2024b) as follows: We use a small calibration dataset (in our case, 256 sequences). When compressing the i-th transformer block, we collect its expected output (()) from the original dense model. We also collect the output of the (i 1)-th block
5

---

Published in Transactions on Machine Learning Research (03/2026)
after compression, which becomes the imput X of the i-th block. Before compression, first fine-tune the block to correct errors from previous blocks. Then we compress the query, value, and output matrices, fine-tune the rest of the block, and compress the remaining matrices. After the compression, we fine-tune the remaining unquantized parameters (scaling vectors in cach binary factorization).
PV-tuning. We also attempt to fine-tune signs in the binary matrices. This is challenging since storing dense weights (e.g,, for straight-through estimation or PV tuning) requires more memory than the dense model. We reduce the memory requirements by always rumning PV-tuning only subset of layers. In each k-step (we set k 50), select random subset of layers to be PV-tuned, where each layer has probability of 1/10 of being selected. We also fine-tune all continuous paramcters in all layers.
3.5 Nonuniform layer compression ratios
Most quantization algorithms quantize all layers to the same bitwidth. In many cases, they offer limited compression sizes (i.e. integer avg. bits per layer) and combining different ones is non-trivial.
Evopress (iieberling et al. 2024 proposes to use evolutionary algorithms to select layer compression ratios, while HIGGS (Malinovskii et al. 2025 shows that for lower compression ratios, one can find a linear relationship between layer-wise compression errors and final model perplexity. While both approaches select good compression ratios, we found that the improvement disappears after fine-tuning.
We propose a different approach based on two observations about DBF. First of all, DBF can have almost any compression ratio (even constraining the middle dimension to multiples of 32 leads to 0.03 bits/ /weightt steps in compression ratio or smaller). Secondly, after we compute DBF for higher compression ratio, we can consider the middle dimension as channels and use the previously proposed channel pruming meth- ods. Specifically, we use the criteria used in Yang et al.] (2023): Molchanov et al.. (2019). Considering our factorization (a A+1 2 (B+1 @ "² for each my we calculate score s; as (we sum over multiple batches):
dE 3m;
2
Σ
St
m,
We then sort scores 8; in all layers with the same size (o.g., in Llama3 we group (k,v), (o,q). (up.gate,down) layers together) and keep only middle channels with the highest scores. This gives us the desired sizes of each layer, and run the compression process again. This naturally gives an iterative compression process.
4 Experiments
4.1 Uniform compression of LLMs
We evaluate our proposed Double binary factorization on compressing Llama2-7B (Touvron et al. 2023 and Llama3-8B GGrattafiori et al. 2024 models.
Calibration and fine-tuning data. We use Redpajama dataset (Weber et al. 2024 for calibration and fine-tuning. We use the same preprocessing as in Malinovskii et al. (2024). We select 256 random sequences as a calibration dataset and use them to compute input activation norms and output gradient norms for each linear layer. We also use for fine-tuning during compression.
Compute resources. Compression with fine-tuning during compression was done on one RTX 4090 GPU and takes 6-8 hours. We further fine- fine-tuning on A100 GPUs for 24 hours.
Evaluation metrics. We evaluate compression methods using Wext2 (Merity et al. 2016) perplexity and zero-shot accuracy on ARC-easy and ARC-challenge (Clark et al., 2018), PIQA (Bisk et al. 2020
and Winogrande (Sakaguchi et al., 2021). We also evaluate Llama3-8B 2-2.3 bit models on MMLU [Hendrycks et al. 2020) and GSMSk 8-shot (Cobbe et al. 2021 benchmarks.
5-shot
Compared methods. We compare against multiple state-of-the-art methods in 1-2.3 bit compression range. We compare with AQLM (Fgiazarian et al., 2024) with PV-tuning (Malinovskii et al. 2024). We only
6

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: Results for uniform compression of Llama2-7B. Avg.
bits refers to the average number of bits per
original weight.
Avg. bits
Method
WikiText ppl.
ArcC
ArcE
PiQA
Wino
Avg. zero shot
16
Dense
5.12
43.43
76.34
78.07
69.06
66.73
AQLM PV
5.84
38.91
72.90
77.37
67.72
64.23
2.4
Caldera
5.84
35.90
64.60
76.50
65.70
60.68
2.3
DBF
5.87
38.39
73.35
76.49
66.85
63. 77
2.3
DBF + PV
5.85
39.33
73.65
76.55
67.08
64.15
2
QTIP
5.86
39.76
73.31
76.27
67.20
64.14
2
QUIP#
6.19
37.79
71.88
75.46
65.66
62.70
2.1
Caldera
6.30
35.40
63.30
75.40
64.60
59.68
2
DBF
6.14
37.03
72.05
76.22
65.82
62.78
2
DBF PV
6.09
36.94
71.88
76.16
66.06
62.76
1.5
AQLM PV
7.32
29.44
64.14
73.12
63.38
57.52
1.5
DBF
7.16
35.06
66.32
73.28
64.40
59.77
1.5
DBF PV
7.01
35.58
67.12
73.72
63.69
60.03
OneBit
9.73
29.61
41.58
68.12
58.41
49.43
1.1
BiLLM
32.48
24.40
36.20
60.60
52.40
43.40
1
DBF
9.57
26.45
55.30
67.41
58.87
52.01
DBF + PV
8.76
27.47
57.87
68.66
58.40
53.10
Table 2: Results for uniform compression of Llama3-8B. Avg. original weight.
bits refer to the average number of bits per
Avg. bits
Method
WikiText ppl
ArcC
ArcE
PiQA
Wino
Avg. zero shot
16
Dense
5.54
50.43
80.09
79.71
72.61
70.71
2.3
AQLM + PV
6.76
42.32
75.46
78.45
71.67
66.98
2.4
Caldera
7.34
42.30
73.60
76.50
70.30
65.68
2.3
DBF
6.97
45.73
76.93
77.47
70.32
67.61
2.3
DBF + PV
6.86
45.56
76.76
78.01
71.90
68.06
2
QTIP
7.33
44.20
75.20
77.60
70.70
66.93
2
QUIP#
7.84
39.20
72.90
75.60
68.20
63.98
2.1
Caldera
8.06
40.00
71.50
76.00
69.50
64.25
2
DBF
7.48
41.04
74.45
76.82
67.95
65.07
2
DBF + PV
7.30
44.45
75.33
76.98
70.08
66.71
1.5
AQLM + PV
9.43
32.68
65.78
72.63
64.40
58.87
1.5
DBF
9.37
33.61
67.59
73.99
64.64
59.96
1.5
DBF PV
9.05
35.58
68.68
74.10
65.43
60.95
1.1
BiLLM
28.80
17.70
36.00
56.10
51.00
40.20
1
DBF
15.61
21.67
50.25
65.17
56.43
48.38
1
DBF + PV
13.57
25.08
56.22
67.46
58.80
51.89
compare against PV-tuned models, which publicly available. We also compare against QulP# [Teng et al.) 2024a), QTIP (Teeng et al. 2024b) and Caldera (Saha al. 2024). Finally, compare against OneBit (u et al. 2024) and BiLLM (Huang et al. 2024), which are specifically designed for extreme 1-bit quantization. For our method, we always report numbers with simple tuning (only continuous parameters) and with PV-tuning.
Results. Results for uniform compression are shown Tab. and 2 with PV-tuning for 2.3 bit compression. We are marginally worse than QTIP but better than QuIP# for bit compression. Both methods require decompression into full precision weight, while DBF mostly
Our results are comparable to AQLM
7

---

Published in Transactions on Machine Learning Research (03/2026)
Table 3: Results for uniform compression of Llama3-8B on MMLU and GSM8K.
Avg.
bits
Method
WikiText ppl
MMLU
GSM8k
16
Dense
5.54
65.29
54.89
2.3
AQLM + PV
6.76
56.20
35.25
2.3
DBF + PV
6.86
56.37
27.59
2
QTIP
7.33
56.20
24.79
2
DBF + PV
7.30
52.67
26.15
additions. We are better than Caldera for everything except Llama2-7B compression in 2.3 bit range. For 1-1.5 bit compression, our method is superior to other tested methods. Note that OneBit was fine-tuned for much longer than DBF (OneBit reports 7 days on 8 H100s, we tune for 1 day on 4 A100s), and DBF with only continuous parameters fine-tuning is still better. Also, note that to achieve the average 1 bit per layer, we need the middle dimension of factorization to be smaller than the dimensions of the original matrix. But even with the low-rank bottleneck, DBF is still better than OneBit.
Results on MMLU and GSM8k. We evaluated the best-performing models for 2-2.3 bit compression of Llama3-8B. Results are in Tab. 3 Results are slightly erratic. DBF comparable AQLM on MMLU, but much worse in GSM8k. On the other hand, QTIP performs very well on MMLUU but worse on GSM8k. Note that these are raw pretrained models, which were not specifically tuned for reasoning.
4.2 Non-uniform compression of LLMs
Due to a lack of computational resources, we run only one round of iterative pruming. We compress Llama3- 8B in the same way as above. We start with DBF with 2.1 bits/ biigh.. We then estimate scores for middle elements of the factorization and compute sizes of each layer. We found that restricting each layer to have at least 1.5 bits/weigtt /weight leads slightly better results. We then repeat the compression and fine-tuning process from the start. We are able to push perplexity for Llama3-8B further down from 7.30 to 7.26. We see that nonuniform distribution gives benefit even with one iteration of redistribution.
4.3 Properties of DBF
Adherence to weight importance. In this experiment, we test whether DBF has lower approximation error for weights with higher importance. We factorize layer 7.sclf _oelf_atttkke attnnk_ttppr from Llama3-8B with imput and output importance calculated from the calibration set as the main experiments. As controls, we use basic 3-bit scalar quantization and OneBit. In case of OneBit, we use importance scaling as described Sec. 3.3 We plot the results in Fig. 2 We see that neither simple scalar quantization (this is expected) nor OneBit can follow weight importance, while DBF decreases error as the weight importance increases.
Approximation error vs avg. number of bits. We also explore how the approximation error changes with the change in compression ratio (avg. number of bits per layer). Again, we compare DBF with scalar quantization and OneBit. This time, we do not use weight importance. We test different compression ratios for DBF and scalar quantization. Results are shown in Fig. 3 We see that for 1-3 bit compression, our method is better. However, at bits and higher, scalar quantization is better than DBF. We found out that we can improve higher bit compression (larger middle sizes) by using "size annealing". We run 80% of iterations with 2-bit compression. And during the last 20% of iterations, dimension (initializing the expanded part with small random parameters). This slightly improves 3,4 and 6-bit compression and suggest that limited performance for higher bit width is algorithmic problem not fundamental limitation of DBF.
gradually expand the middle
Scaling to bigger matrices. We investigate scaling DBF to larger matrices present even larger language models. We track relative apx. error for different matrices present in Llama3-8B, Llama3-700 and Llama3.1 3. 405B models. Results shown in Fig. 3 We do not see any degradation for larger matrices.
8

---

Published in Transactions on Machine Learning Research (03/2026)
DBF
3-bit scalar quant
OneBit
0.0020
0.00045
0.00045
Weight apx. error
0.0015
0.00030
0.00030
0.0010
0.00015
0.00015
0.0005
0.00000
0.00000
0.0000
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
Weight importance
Weight importance
Weight importance
Comparison of weight importance (calculated as a product of input and output importance) vs. approximation error on 7.self attn.k proj of Llama3-8B. Blue points are actual matrix elements, and the orange line is smoothed version.
Figure 2:
OneBit
DBF
DBF w/ size annealing
Scalar quant
L3-8B-9.v __v__roj
L3-8B-100 g0joproo__pp
Rei apx. error
10-1
10-1
10-2
10-2
10-3
2
6
2
L3-70B-38.Q_proj
L3.1-4058-63.g
Rei apx. error
10-1
10-1
10-2
10-2
10
2
Number of bits
Number of bits
Figure 3: Comparison of number of bits vs. approximation error on two layers of Llama3-88 8B and on selected proj layers of bigger Llama3 models. Compression to 4 and higher number of bits threw OOM error on 40GB GPU for 405B Llama.

---

Published in Transactions on Machine Learning Research (03/2026)
Table 4: Comparison of speed of the FP16 matrix-vector multiplication in PyTorch DBF for various matrix sizes Nvidia RTX 4090.
Avg.
bits
4096 x 4096
4096 x 14336
8192 x 8192
8192 x 28672
Original
16
61 μas
153 μas
200 μus
548 pas
DBF
2.3
29 μs (x2.14)
56 μus (x2.71)
64 ju (x3.11)
167 μss (x3.25)
DBF
2
uu8 (x2.31)
jus (x2.98)
58 jas (x3.44)
149 jss (x3.65)
DBF
1.5
24 js (x2.61)
43 jus (x3.57)
51 js (x3.93)
115 ps (x4.71)
DBF
1
20 ps (x3.01)
37 μus (x4.17)
38 jas (x5.31)
84 js (x6.52)
Table 5: Batch size 1 decoding throughput on Nvidia RTX 4090 for two versions of Llama models.
Avg.
bits
2-7B
3-8b
Original
16
68 tok/s
60 tok/s
DBF
2.3
144 tok/s (x2.12)
121 tok/s (x2.01)
DBF
2
153 tok/s (x2.25)
133 tok/s (x2.22)
DBF
1.5
167 tok/s s (x2.46)
153 tok/s (x2.55)
DBF
170 tok/s / (x2.50)
174 tok/s /8 (x2.90)
4.4 Inference speed
Although main objective is to show that achieve competitive network compression using just binary matrices, we also obtain practical speedups on current hardware. We benchmark matrix-vector multiplication and decoding speeds of LLMs with batch size 1. To perform the computation, we use the implementation of binary matrix multiplication from the package gemlite (Badri Shaji 2023
We report results of matrix-vector vector multiplication speed for various matrix sizes found in typical LLMs in Tab. DBF is 2-3.5x faster with 2 bits/ bits/weights and 3-6x faster with bit/weight.
For LLM decoding, measure the time it takes to generate 128 tokens from emply prompt, performed compiled computational graphs, with batch size 1, and report the average number of generated tokens per second on a single GPU. We report results Tab. DBF achieves -2.0-2.9x speedup compared to the fp16 bascline.
Although our empirical throughput numbers were timed on NVIDIA GPUs, DBF can be fast on a broad class of accelerators due to its flexibility and simplicity.
5 Conclusions and Future Work
In this paper, we introduced Double Binary Factorization (DBF), novel compression method that ap- proximates weight matrices as the product of two binary (sign) matrices, each accompanied by appropriate scaling vectors. The main benefit of DBF is replacing energy-costly multiplications with cheaper additions. DBF extends the concept. of binary quantization, achieving competitive or superior compression performance compared to srate-of-thcarr quantization approaches. We demonstrated that DBF significantly outperforms existing methods using binary weight matrices and achieves comparable accuracy with state-of-the-art meth- ods at compression rates of around 2 bits per weight. Furthermore, approach is also practical, yielding speedups of 2-3.5x relative to dense baselines and promising significant energy savings due to the computa- tional simplicity of binary operations, which only use additions instead of multiplications.
A distinct advantage of DBF its inherent flexibility. Unlike many quantization approaches, DBF allows continuous control over compression ratios by adjusting the factorization's middle dimension. We further leveraged this flexibility by introducing an iterative, importance-driven pruning approach to dynamically allocate compression budgets across layers, enhancing overall compression efficacy.
10

---

Published in Transactions on Machine Learning Research (03/2026)
There multiple directions for future research to address DBF limitations. One of the main limitations of DBF is the problematic fine-tuning of binary matrices. While PV-tuning brings some benefits, we think that this can be further improved by doing factorization on-the-fly during fine-tuning. Furthermore, inte- grating the iterative pruning scheme with fino-tuning would also be beneficial and allow simplification of the compression process.
Acknowledgments
This rescarch was supported by funding from the Slovak Rescarch and Development Agency grant APVV- 24-0045 and by grants 1/0140/25, and 1/0538/22 from the Slovak research grant agency VEGA. Part of the research results was obtained using the computational resources procured in the national project National competence centre for high performance computing (project code: 311070AKF2) funded by European Re- gional Development Fund, EU Structural Funds Informatization of society, Operational Program Integrated Infrastructure. We acknowledge the EuroHPC Joint Undertaking for awarding this project access the Fu- roHPPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through an EuroHPC Benchmark and AI Access calls.
References
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication sgd via gradient quantization and encoding. Advances in neural information processing systems, 30, 2017.
Hicham Badri and Appu Shaji. Half-quadratic quantization of large machine learning models, November 2023. URL
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432 7439, 2020.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Pelcato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundalions and Trends@ in Machine learning, 311):1-122, 2011.
Vladimír Boža. Fast and effective weight update for pruned large language models. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL
Vladimír Boža and Vladimír Macko. Two sparse matrices are better than one: Sparsifying neural networks with double sparse factorization. In 13th International Conference on Learning Representations, ICLR 2025, 2025.
Daria Cherniuk, Stanislav Abukhovich, Anh-Huy Phan, Ivan Oseledets, Andrzej Cichocki, and Julia Gusak. Quantization aware factorization for deep neural network compression. Journal of Artificial Intelligence Research, 81:973 988, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. orXiv preprint arXiv:1803.05A. 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Malthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 168, 2021.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. arXie preprint arXiv:14122 7024, 2014.
Vage Egiazarian, Andrei Panfcrov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. In forty-forst International Conference Machine Learning, 2024.
11

---

Published in Transactions on Machine Learning Research (03/2026)
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10333 PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-trainin quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint or 0rXi0:2407.21788 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazcika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, arXic preprint urXiv:2009.083: 03300, 2020.
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiao- juan Qi. Billm: Pushing the limit of post-training quantization for llms. aix preprinl (rXiv:2402.04221, 2024.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient -eitee-- only inforence. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2704 2713, 2018.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. orXiv preprinl arXiv:1405.3866, 2014.
Chong Li and CJ Shi. Constrained optimization based low-rank approximation of deep neural networks. In Proceedings of the European Conference on Compuler Vision (ECCV), pp. 732-747, 2018.
Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Al- istarh, and Peter Richtarik. Pv-tuning: Beyond straight-through estimation for extreme Ilm compression. arXiv preprint ar Xiv::2405. 14852, 2024.
Vladimir Malinovskii, Andrei Panferov, Ivan Ilin, Han Guo, Peter Richtárík, and Dan Alistarh. Higgs: Pushing the limits of large language model quantization via the linearity theorem. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 18557-0886,, 2025.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint ar crXi::6609.022 07843,, 2016.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11264-11272, 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 225-542. Springer, 2016.
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. Compressing large lan- guage models using low rank and low precision decomposition. Advances in Neural Information Processing Systems, 37:88981 89018, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge al scale. Communications of the ACM, 64(9):99 106, 2021.
Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic, and Dan Alistarh. Evopress: Towards optimal dynamic model compression via evolutionary search. arXiv preprint ar arXiv:24 1000440.1...0 14649, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint Xrxiv:2306. 1 11695, 2023.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXic preprint orXiv:2307.09988. 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip # Even better Ilm quantization with hadamard incoherence and lattice codebooks. In Forty-first International Conference Muchine Learning, 2024a.
Albert Tseng, Qingyao Sun, David Hou, and Christopher M De Sa. Qtip: Quantization with trellises and incoherence processing. Advonces in Neural Informalion Processing Systems, 37:59597-5990 596200 2024b.
Hongyu Wang, Shurming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.114 11455 2023.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-awaar singular value decomposi- tion for large language model compression. arXiv preprint arXiv:2403.0730 2024.
Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. Svd-Ilm v2: Optimizing singular value truncation for large language model compression. arXiv preprinl wxXx:22503.13444 2025.
Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, al. Redpajama: an open dataset for training large language models. arXiv preprint 1rXiv:24111999 2024.
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision trans- former pruming with hessian-aware saliency. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognation, pp. 18547-18557, 2023.
13


Reviews:
[{"review_id": "I5A1DJYVd0", "reviewer": "Reviewer_7mr1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on binary quantization methods used in Large Language Model (LLM) compression. The authors propose a method called Double Binary Factorization (DBF) in order to achieve superior compression to Single Binary factorization and rates that claim to rival existing binarization/quantization approaches. The approach is flexible and allows for continuous control over compression ratios. The authors propose an algorithm for DBF and provide a numerical study comparing the compression and approximation errors with other state-of-the art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors largely claim that: 1) DBF can improve compression performance due to it's increased flexibility over other methods and 2) DBF may be performed faster (or take advantage of hardware specialized for accelerated computation) due to the binary matrices that require only additions. This work has numerical results that generally support both claims.", "audience_yn": "Yes", "audience_explanation": "I believe this work has broad interest to researchers in data compression and information theory. It would also appeal to researchers in theoretical machine learning, as the method serves as a benchmark for matrix (and tensor) compression techniques in LLMs.", "requested_changes": "**My comments can be categorized into two major areas:**\n\n1. General format and readability\na) This work focuses on a special category of neural networks called large language models and the authors frequently reference neural network terminology such as “layers”, “weight matrices” and “activation function” without properly defining them. A simple figure defining a general LLM with weight matrices and activation functions  would produce a clearer and more engaging introduction to a broader audience. It will also allow the reader to better understand Figure 1., as well as the differences of the state-of-the-art methods in the current literature.\n\n2. Experimental results and discussion:\n\na) Table 1 and Table 2 indicate that DBF allows for considerable improvements in the 1-1.5 bit regime. In the 2-2.3 bit regime, the authors claim DBF to be comparable to AQLM and better than QUIP#. In fact, DBF also seems comparable to QUIP# and the improvement in performance seems marginal. What I think would help the authors and the quality of the work is if the work contained the answers to the following questions: \n\t*Under which application or regime would a practitioner be interested in 1 bit uniform compression? Why is such a compression useful?*\n\nb) Concern with figure 2: the impact of DBF on approximation error is unclear. I agree that the figure shows that DBF has lower approximation error for weights with higher importance. However, with all methods, the approx error (irrespective of weight importance) is at most $2\\times10^-3$ (and smoothed version is always less than $5\\times10^-4$), which is already a low error. \n\t*How are you mathematically defining weight importance? How are you mathematically defining approximation error?*\n\nc) I believe an expanded discussion on the enhanced computational speed/complexity of DBF compared to other methods is needed. The work claims this to be a benefit of DBF, where it can replace costly multiplications with cheaper additions. When considering the tradeoff between accuracy and speed, some practitioners/applications may tolerate slightly lower accuracy if computational costs are significantly reduced. One table quantifying DBF speed compared to PTQ methods would be sufficient and would strengthen the discussion for the experimental results in Table 1 and 2.\n\n**Minor issues:**\n1. QuIP# and QTIP , GPTQ, BiLLM stated throughout text but not defined.\n2. Weight Binarization is not formally defined. BitNet and Onebit are referenced without being explicitly defined as weight binarization approaches. \n3. Minor typo on page 4: “while ADMM provably...”", "broader_impact_concerns": "I do not see any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gBfhSPcNLt", "reviewer": "Reviewer_G7Du", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Large Language Models demand significant memory and computation resources. Standard binary quantization reduces these costs but often severely degrades model accuracy. To address this, the authors propose Double Binary Factorization (DBF). This method factorizes dense weights into the product of two binary sign matrices and floating-point scaling vectors. This approach replaces expensive multiplications with additions and enables flexible compression ratios by adjusting the inner dimension size. The authors evaluate DBF on Llama2 and Llama3 models using language modeling and zero-shot reasoning tasks. DBF outperforms OneBit in 1-bit perplexity and remains competitive with state-of-the-art methods in the 2-bit regime while achieving inference speedups of 2-3.5x on standard GPUs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the claims are generally well-supported, though the scope of the evidence has some notable gaps:\n\n- The claim of superior accuracy in extreme compression is convincingly supported by experiments showing DBF achieves lower perplexity (9.57) than the OneBit baseline (9.73) on Llama2-7B. \n\n- Assertions regarding practical efficiency are substantiated by hardware benchmarks on an RTX 4090, which demonstrate clear 2-3.5x inference speedups over dense baselines by utilizing addition-only operations.\n\n- The method's flexibility is accurately validated through ablation studies showing that adjusting the factorization's intermediate dimension allows for fine-grained, fractional bit-width targets (e.g., 1.5 bits) unlike standard integer quantization.\n\n- However, the evidence is not fully comprehensive as the evaluation relies solely on perplexity and simple zero-shot tasks, omitting complex reasoning benchmarks (like GSM8K) or any computer vision tasks despite the broad title.", "audience_yn": "Yes", "audience_explanation": "Yes. The findings are likely to be of significant interest to several sub-communities within the TMLR audience, particularly those focused on efficient machine learning and Large Language Model (LLM) deployment.", "requested_changes": "- Evaluation on Complex Benchmarks : The current evaluation relies primarily on Perplexity and simple zero-shot tasks (ARC, PiQA). To demonstrate that extreme quantization does not disproportionately harm complex reasoning capabilities, please evaluate the method on MMLU (Massive Multitask Language Understanding) and GSM8K (Grade School Math). These are standard \"stress tests\" for quantization that often reveal brittleness not captured by WikiText perplexity. (Required)\n\n- Clarification of Scope or Title The paper is titled \"Compressing neural networks...\", yet all experiments are restricted exclusively to text-only Large Language Models (LLMs). Please address this discrepancy by doing one of the following:\n\n- Modify the title to reflect the specific scope (e.g., \"Compressing Large Language Models...\").\n\n- (Optional but encouraged) Include a preliminary experiment on a VLM (e.g., Llama 3.2 Vision or LLaVA). Since the method is already optimized for Llama, demonstrating it on a multimodal Llama variant would be a natural and compelling way to justify the broader \"Neural Networks\" claim without switching architectures entirely. Note: This experiment is not required for acceptance, but would significantly strengthen the paper's claims.", "broader_impact_concerns": "I do not see any broader concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0kJZDxbUdg", "reviewer": "Reviewer_qhJ2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Double Binary Factorization (DBF), a post-training compression method that decomposes dense LLM weight matrices into a product of two binary sign matrices (±1) and three scaling vectors. The work is an extension to their prior ICLR 2025 submission (https://arxiv.org/pdf/2409.18850). By utilizing an ADMM-based heuristic to solve the non-convex factorization problem, the authors achieve competitive performance in the 1-2 bits per weight regime.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The evidence for DBF’s effectiveness is clear and well-documented.\n\n* Extreme Compression: DBF significantly outperforms OneBit and BiLLM in 1-bit settings.\n* Speed: The method provides tangible 2-3.5x speedups on current hardware by replacing multiplications with additions.", "audience_yn": "Yes", "audience_explanation": "This work is of relevance to the journal's audience, particularly those focused on efficient machine learning and LLM deployment. As the scale of models grows, the transition from multiplication-heavy to addition-based inference is a significant research frontier. DBF offers a practical bridge between extreme binarization and high-precision quantization.", "requested_changes": "My main comments are around two topics: contextual positioning & scalability.\n\n**Contextual Positioning**: While the empirical results are strong, the paper’s discussion of related work is somewhat narrow. It misses several \"joint factorization and quantization\" approaches that provide necessary context:\n    \n1. Quantization Aware Factorization (Gusak et al., 2024): This is a significant omission. Like the current submission, it uses ADMM for joint factorization (specifically CP-decomposition) and quantization to reduce approximation error. A comparison or acknowledgement of how DBF’s binary constraint differs from their integer-grid constraint would be beneficial.\n\n2. CALDERA (Saha et al., 2024): Recent work on approximating weights via $W \\approx Q+LR$ (quantized backbone + low-rank factors) is highly relevant for the 2-bit regime where DBF claims competitiveness.\n\n3. SVD-LLM & SVD-LLM V2 (Wang et al., 2025): The paper would benefit from discussing these truncation-aware SVD methods, which represent the current state-of-the-art for post-training factorization without binary constraints.\n\n**Scalability**: While the results are strong, the paper acknowledges that at 4 bits and higher, standard scalar quantization still outperforms DBF. Additionally, the fine-tuning of binary matrices (PV-tuning) remains memory-intensive and complex. Further investigation into whether the higher-bit limitation is fundamental or algorithmic would strengthen the work.\n\n\n\nMinor comments:\n\n* Page 2: \"...low-rank factorizations come with severe degradation in accura \" -> accuracy.\n* Page 3: \"...we will select kask $k=b \\frac{nm}{n+m}$\"\n* Page 5: \"We use slight variation of the procedure...\". It should be: \"We use a slight variation...\".\n* Page 6: \"...multiple state-to-art methods...\" -> \"state-of-the-art\"\n* Page 9: \"...generate 128 tokens from emtpy prompt...\". empty\n​", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RPqtuGruks", "reviewer": "Reviewer_krJk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Double Binary Factorization (DBF), a novel matrix compression scheme that decomposes each weight matrix into a product of two binary sign matrices combined with learnable scaling vectors. By adjusting the size of the intermediate dimension, DBF provides continuous control over the effective compression rate, enabling fine-grained tuning of the average bits per weight—an advantage over conventional quantization methods that support only discrete bitwidth options. Empirically, DBF delivers competitive perplexity across a broad range of compression levels (1–2.3 bits/weight), outperforming prior 1-bit binarization approaches and achieving accuracy comparable to state-of-the-art quantizers at ~2 bits/weight, while enabling faster inference by replacing multiplications with additions. The authors further propose an importance-aware pruning mechanism for non-uniform per-layer compression.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper has detailed experiment on wikitext to support the idea that their method is general enough to perform well on different degrees of compression. It can be adapted to different devices to balance the computation cost and the performance.", "audience_yn": "Yes", "audience_explanation": "In general, the deployment of large scale model is heavily dependent on the quantization methods.", "requested_changes": "It is desirable to see the scaling property such as when we increase the hidden dimension / depth, what's the performance scaling property on some small-scale task or synthetic task.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)