Repository: openvinotoolkit/nncf
Pull request #3497: [ONNX] Weight compression for opset < 21
Author: login: andrey-churkin
Labels: name: NNCF ONNX
description: Pull requests that updates NNCF ONNX
name: NNCF PTQ
description: Pull requests that updates NNCF PTQ
name: Code Freeze
Created: 2025-05-16T07:44:29Z | Merged: 2025-05-27T11:38:36Z

### Changes

Replace the `MatMul` operation with the `MatMulNBits` operation from ONNX Runtime contrib operators.
See https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md

### Reason for changes

The `DequantizeLinear` operation supports the `block_size` attribute only starting from opset 21.

### Related tickets

Ref: 163946

### Tests

<!--- How was the correctness of changes tested and whether new tests were added -->



Diff:
diff --git a/.ci/cspell_dict.txt b/.ci/cspell_dict.txt
index f9d41a98526..873d248b0cc 100644
--- a/.ci/cspell_dict.txt
+++ b/.ci/cspell_dict.txt
@@ -165,6 +165,7 @@ googlenet
 gptq
 graphdef
 groupnormalization
+groupsize
 groupwise
 grusequence
 gscale
@@ -234,6 +235,7 @@ lyalyushkin
 mapillary
 maskrcnn
 mathbb
+matmulnbits
 matmuls
 maxcolwidths
 maxpool
@@ -316,6 +318,7 @@ openvino
 openvinotoolkit
 oplambda
 opset
+opsets
 outgate
 ovawq
 overcalled
diff --git a/nncf/onnx/graph/model_transformer.py b/nncf/onnx/graph/model_transformer.py
index 60dd069a801..889cf843a59 100644
--- a/nncf/onnx/graph/model_transformer.py
+++ b/nncf/onnx/graph/model_transformer.py
@@ -481,3 +481,15 @@ def remove_initializer(initializer_name: str, model: onnx.ModelProto) -> None:
     original_initializer = next((init for init in model.graph.initializer if init.name == initializer_name), None)
     if original_initializer is not None:
         model.graph.initializer.remove(original_initializer)
+
+
+def remove_node(node_name: str, model: onnx.ModelProto) -> None:
+    """
+    Removes the node from the ONNX model.
+
+    :param node_name: Name of the node to remove.
+    :param model: ONNX model.
+    """
+    original_node = next((node for node in model.graph.node if node.name == node_name), None)
+    if original_node is not None:
+        model.graph.node.remove(original_node)
diff --git a/nncf/onnx/graph/onnx_helper.py b/nncf/onnx/graph/onnx_helper.py
index fcc9f5101db..ec64a41baca 100644
--- a/nncf/onnx/graph/onnx_helper.py
+++ b/nncf/onnx/graph/onnx_helper.py
@@ -327,3 +327,45 @@ def pack_4_bits(tensor: np.ndarray) -> np.ndarray:
     packed_tensor = packed_tensor.reshape(-1, 2)
     packed_tensor = packed_tensor[..., 1::2] << 4 | packed_tensor[..., ::2] & 15
     return packed_tensor
+
+
+def pack_int4_to_uint8(weight: np.ndarray, block_size: int, signed: bool) -> np.ndarray:
+    """
+    Returns `weight` that is stored as uint8 with shape (N, n_blocks_per_col, blob_size) in which:
+        - n_blocks_per_col = CeilDiv(K, block_size)
+        - blob_size = CeilDiv(block_size * bits, 8)
+        - bits = 4
+
+    See https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftmatmulnbits
+    for more details.
+
+    :param weight: A 2D array of shape (K, N) quantized with 4 bits.
+    :param block_size: Number of groupsize used for weight quantization.
+    :param signed: True if the weight has type int4, and False if uint4.
+    :return: A packed weight that can be used as `B` input for `com.microsoft.MatMulNBits` operation.
+    """
+    ceil_div = lambda a, b: (a + b - 1) // b
+    bits = 4  # Number of bits used for weight quantization
+
+    K, N = weight.shape
+    n_blocks_per_col = ceil_div(K, block_size)
+    blob_size = ceil_div(block_size * bits, 8)
+
+    packed_weight = np.zeros((N, n_blocks_per_col, blob_size), dtype=np.uint8)
+
+    for n in range(N):
+        for block_num in range(n_blocks_per_col):
+            begin = block_num * block_size
+            end = min(begin + block_size, K)
+            block = weight[begin:end, n]
+
+            for packed_idx, i in enumerate(range(0, len(block), 2)):
+                # 2 x 4bit is stored in one uint8
+                val_1 = block[i] + 8 if signed else block[i]
+                if i + 1 < len(block):
+                    val_2 = block[i + 1] + 8 if signed else block[i + 1]
+                    packed_weight[n, block_num, packed_idx] = ((val_2 & 0x0F) << 4) | (val_1 & 0x0F)
+                else:
+                    packed_weight[n, block_num, packed_idx] = val_1 & 0x0F
+
+    return packed_weight
diff --git a/nncf/onnx/quantization/quantize_model.py b/nncf/onnx/quantization/quantize_model.py
index 9dfabacdc8c..c8382f43a9a 100644
--- a/nncf/onnx/quantization/quantize_model.py
+++ b/nncf/onnx/quantization/quantize_model.py
@@ -316,9 +316,6 @@ def compress_weights_impl(
     if model.opset_import[0].version < 13:
         msg = "ONNX models with opset version < 13 do not support per-channel quantization."
         raise nncf.ValidationError(msg)
-    if model.opset_import[0].version < 21 and group_size > 0:
-        msg = "ONNX models with opset version < 21 do not support block-wise quantization."
-        raise nncf.ValidationError(msg)
 
     external_data_dir = get_external_data_dir(advanced_parameters)
     external_data_dir = check_external_data_location(model, external_data_dir)
diff --git a/nncf/quantization/algorithms/weight_compression/onnx_backend.py b/nncf/quantization/algorithms/weight_compression/onnx_backend.py
index a962cf163bc..2bac7048639 100644
--- a/nncf/quantization/algorithms/weight_compression/onnx_backend.py
+++ b/nncf/quantization/algorithms/weight_compression/onnx_backend.py
@@ -17,6 +17,7 @@
 
 import nncf
 import nncf.onnx.graph.metatypes.onnx_metatypes as metatypes
+import nncf.tensor.functions as fns
 from nncf.common.graph import NNCFGraph
 from nncf.common.graph import NNCFNode
 from nncf.common.graph.operator_metatypes import OperatorMetatype
@@ -31,6 +32,7 @@
 from nncf.onnx.graph.metatypes.groups import CONVOLUTION_METATYPES
 from nncf.onnx.graph.metatypes.groups import MATMUL_METATYPES
 from nncf.onnx.graph.model_transformer import remove_initializer
+from nncf.onnx.graph.model_transformer import remove_node
 from nncf.onnx.graph.model_transformer import set_initializer
 from nncf.onnx.graph.node_utils import get_weight_quantization_axis
 from nncf.onnx.graph.onnx_helper import ONNX_DTYPE_TO_NNCF_DTYPE
@@ -39,6 +41,7 @@
 from nncf.onnx.graph.onnx_helper import get_tensor
 from nncf.onnx.graph.onnx_helper import get_tensor_value
 from nncf.onnx.graph.onnx_helper import pack_4_bits
+from nncf.onnx.graph.onnx_helper import pack_int4_to_uint8
 from nncf.onnx.graph.transformations.commands import ONNXTargetPoint
 from nncf.parameters import CompressionFormat
 from nncf.parameters import CompressWeightsMode
@@ -46,6 +49,7 @@
 from nncf.quantization.algorithms.weight_compression.backend import WeightCompressionAlgoBackend
 from nncf.quantization.algorithms.weight_compression.config import WeightCompressionParameters
 from nncf.quantization.algorithms.weight_compression.lora_correction import LoraCorrectionAlgorithm
+from nncf.quantization.algorithms.weight_compression.weight_lowering import CompressedWeight
 from nncf.quantization.algorithms.weight_compression.weight_lowering import compress_weight
 from nncf.tensor import Tensor
 from nncf.tensor.definitions import TensorDataType
@@ -75,34 +79,40 @@ def _get_weight_dtype(self, mode: CompressWeightsMode) -> onnx.TensorProto.DataT
             raise nncf.ParameterNotSupportedError(msg)
         return dtype
 
-    def _preprocess_compressed_weight_shapes(
-        self, compressed_weight: Tensor, weight_shape: tuple[int], dequantize_block_size: int
+    def _preprocess_compressed_weight(
+        self,
+        compressed_weight: CompressedWeight,
+        weight_shape: tuple[int],
+        dequantize_block_size: Optional[int] = None,
+        apply_transpose: bool = False,
     ) -> tuple[np.ndarray, np.ndarray, Optional[np.ndarray]]:
         """
-        Helper function to preprocess the tensor shapes for the compressed weight tensors to ONNX shapes expectations.
-        The function reshapes the weight tensor and squeezes the scale and zero point tensors based on the
-        dequantize_block_size parameter.
+        Preprocess compressed weight tensor to ONNX-compatible form.
 
-        :param compressed_weight: The compressed weight tensor.
-        :param weight_shape: The original shape of the weight tensor.
-        :param dequantize_block_size: The block size for dequantization.
+        :param compressed_weight: Compressed weight struct.
+        :param weight_shape: Target shape for the weight tensor.
+        :param dequantize_block_size: If given, affects squeezing shape for scale and zero_point.
+        :param apply_transpose: Whether to transpose scale and zero_point.
         :return: A tuple containing the reshaped weight tensor, scale tensor, and zero point tensor (if applicable).
         """
-        weight_tensor = compressed_weight.tensor
-        weight_tensor = weight_tensor.reshape(weight_shape)
+        tensor = compressed_weight.tensor.reshape(weight_shape)
         scale = compressed_weight.scale
         zero_point = compressed_weight.zero_point
+
+        axis = 1 if dequantize_block_size else None
+        scale = scale.squeeze(axis=axis)
         if zero_point is not None:
-            zero_point = zero_point.astype(weight_tensor.dtype)
-        if dequantize_block_size:
-            scale = scale.squeeze(axis=1)
-            if zero_point is not None:
-                zero_point = zero_point.squeeze(axis=1)
-        else:
-            scale = scale.squeeze()
+            zero_point = zero_point.squeeze(axis=axis)
+
+        if apply_transpose:
+            scale = fns.transpose(scale)
             if zero_point is not None:
-                zero_point = zero_point.squeeze()
-        return weight_tensor.data, scale.data, zero_point.data if zero_point is not None else None
+                zero_point = fns.transpose(zero_point)
+
+        if zero_point is not None:
+            zero_point = zero_point.astype(tensor.dtype)
+
+        return tensor.data, scale.data, zero_point.data if zero_point is not None else None
 
     @property
     def matmul_metatypes(self) -> list[OperatorMetatype]:
@@ -198,6 +208,8 @@ def transform_model(
         advanced_parameters: AdvancedCompressionParameters = AdvancedCompressionParameters(),
     ) -> onnx.ModelProto:
         self._check_arguments_for_transform_model(lora_correction_algo, compression_format, advanced_parameters)
+        opset_version = model.opset_import[0].version
+
         for wc_params in weight_compression_parameters:
             compression_config = wc_params.compression_config
             node = wc_params.node_with_weight
@@ -210,23 +222,43 @@ def transform_model(
                 None if precomputed_zero_points is None else precomputed_zero_points.get(wc_params.weight_name),
             )
             dequantize_block_size = max(compression_config.group_size, 0)  # 0 - is no block wise quantization
-            compressed_weight, scale, zero_point = self._preprocess_compressed_weight_shapes(
-                compressed_weight, weight.shape, dequantize_block_size
-            )
-
             dequantize_axis = (
                 get_weight_quantization_axis(node, wc_params.weight_port_id) if dequantize_block_size <= 0 else 0
             )  # axis = 0 when blockwise
-            self._add_dequantize_linear_layer(
-                model,
-                compressed_weight,
-                scale,
-                zero_point,
-                dequantize_axis,
-                dequantize_block_size,
-                wc_params.weight_name,
-                self._get_weight_dtype(compression_config.mode),
-            )
+
+            # NOTE: The `DequantizeLinear` operation supports the `block_size` attribute only starting from opset 21.
+            # For opsets earlier than 21, we use the `MatMulNBits` operation from ONNX Runtime contrib operators.
+            # See https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md
+            if opset_version < 21 and dequantize_block_size > 0:
+                compressed_weight, scale, zero_point = self._preprocess_compressed_weight(
+                    compressed_weight, weight.shape, dequantize_block_size=None, apply_transpose=True
+                )
+                self._replace_matmul_with_matmulnbits(
+                    model,
+                    wc_params,
+                    compressed_weight,
+                    weight,
+                    scale,
+                    zero_point,
+                    dequantize_axis,
+                    dequantize_block_size,
+                    wc_params.weight_name,
+                    self._get_weight_dtype(compression_config.mode),
+                )
+            else:
+                compressed_weight, scale, zero_point = self._preprocess_compressed_weight(
+                    compressed_weight, weight.shape, dequantize_block_size=dequantize_block_size
+                )
+                self._add_dequantize_linear_layer(
+                    model,
+                    compressed_weight,
+                    scale,
+                    zero_point,
+                    dequantize_axis,
+                    dequantize_block_size,
+                    wc_params.weight_name,
+                    self._get_weight_dtype(compression_config.mode),
+                )
         return model
 
     @staticmethod
@@ -301,13 +333,21 @@ def _add_dequantize_linear_layer(
             )
             new_initializers.append(zero_point_initializer)
 
-        dequantize_node = helper.make_node(
-            "DequantizeLinear",
-            inputs=deq_inputs,
-            outputs=[dequantized_weight_output],
-            block_size=block_size,
-            axis=axis,
-        )
+        if block_size != 0:
+            dequantize_node = helper.make_node(
+                "DequantizeLinear",
+                inputs=deq_inputs,
+                outputs=[dequantized_weight_output],
+                block_size=block_size,
+                axis=axis,
+            )
+        else:
+            dequantize_node = helper.make_node(
+                "DequantizeLinear",
+                inputs=deq_inputs,
+                outputs=[dequantized_weight_output],
+                axis=axis,
+            )
 
         # Add the node and initializers to the model
         model.graph.initializer.extend(new_initializers)
@@ -328,3 +368,94 @@ def _add_dequantize_linear_layer(
         # Update the node mapping
         self.name_to_node_map[dequantize_node.name] = dequantize_node
         return model
+
+    def _replace_matmul_with_matmulnbits(
+        self,
+        model: onnx.ModelProto,
+        weight_compression_parameters: WeightCompressionParameters,
+        quantized_weights: np.ndarray,
+        orig_weight,
+        scale: np.ndarray,
+        zero_point: Optional[np.ndarray],
+        axis: int,
+        block_size: int,
+        weight_name: str,
+        weight_dtype: onnx.TensorProto.DataType,
+        scale_dtype: onnx.TensorProto.DataType = onnx.TensorProto.FLOAT,
+    ):
+        if weight_dtype == onnx.TensorProto.INT4:
+            quantized_weights = pack_int4_to_uint8(quantized_weights, block_size, signed=True)
+        elif weight_dtype == onnx.TensorProto.UINT4:
+            quantized_weights = pack_int4_to_uint8(quantized_weights, block_size, signed=False)
+
+        quantized_weight_name = weight_name + "_quantized"
+        scale_name = weight_name + "_scale"
+        zero_point_name = weight_name + "_zero_point"
+
+        quantized_weights_initializer = onnx.helper.make_tensor(
+            quantized_weight_name,
+            onnx.TensorProto.UINT8,
+            quantized_weights.shape,
+            quantized_weights.tobytes(),
+            raw=True,
+        )
+
+        # Create initializers for the quantized weights, scale, and zero point
+        scale_initializer = numpy_helper.from_array(
+            np.array(scale, dtype=helper.tensor_dtype_to_np_dtype(scale_dtype)), name=scale_name
+        )
+        new_initializers = [quantized_weights_initializer, scale_initializer]
+        if zero_point is not None:
+            zero_point_initializer = onnx.helper.make_tensor(
+                zero_point_name, onnx.TensorProto.UINT8, zero_point.shape, zero_point.tobytes(), raw=True
+            )
+            new_initializers.append(zero_point_initializer)
+
+        original_matmul = self.name_to_node_map[weight_compression_parameters.node_with_weight.node_name]
+
+        activation_input_name = None
+        for input_name in original_matmul.input:
+            if input_name != weight_name:
+                activation_input_name = input_name
+        assert activation_input_name is not None, "Activation input name not found in original matmul node"
+
+        # Create MatMulNBits
+        inputs = [activation_input_name, quantized_weight_name, scale_name]
+        if zero_point is not None:
+            inputs.append(zero_point_name)
+
+        K, N = orig_weight.shape[0], orig_weight.shape[1]
+        matmul_n_bits = helper.make_node(
+            op_type="MatMulNBits",
+            inputs=inputs,
+            outputs=[original_matmul.output[0]],
+            K=K,
+            N=N,
+            accuracy_level=0,
+            bits=weight_compression_parameters.compression_config.num_bits,
+            block_size=weight_compression_parameters.compression_config.group_size,
+            domain="com.microsoft",
+            name=original_matmul.name + "_compressed_matmul",
+        )
+
+        # Add the node and initializers to the model
+        model.graph.initializer.extend(new_initializers)
+
+        # Insert the MatMulNBits node before the consumer nodes
+        insert_index = len(model.graph.node)
+
+        for i, node in enumerate(model.graph.node):
+            for j, input_name in enumerate(node.input):
+                if input_name == original_matmul.name:
+                    insert_index = min(insert_index, get_node_index(model, node.name))
+                    node.input[j] = matmul_n_bits
+
+        # Insert the MatMulNBits node before the first consumer node
+        model.graph.node.insert(insert_index, matmul_n_bits)
+        # Remove original weight initializer
+        remove_initializer(weight_name, model)
+        # Remove original matmul
+        remove_node(original_matmul.name, model)
+        del self.name_to_node_map[original_matmul.name]
+        # Update the node mapping
+        self.name_to_node_map[matmul_n_bits.name] = matmul_n_bits
diff --git a/tests/onnx/quantization/test_weights_compression.py b/tests/onnx/quantization/test_weights_compression.py
index 2d90c65b3cb..6e019a53e96 100644
--- a/tests/onnx/quantization/test_weights_compression.py
+++ b/tests/onnx/quantization/test_weights_compression.py
@@ -18,7 +18,6 @@
 from onnx import numpy_helper
 from onnxruntime import InferenceSession
 
-import nncf
 from nncf import CompressWeightsMode
 from nncf.onnx.graph.onnx_helper import get_edge_shape
 from nncf.onnx.graph.onnx_helper import get_tensor
@@ -267,23 +266,22 @@ def test_compression_with_inference(mode):
     session.run(None, {"input": input_data})
 
 
-@pytest.mark.parametrize(
-    "opset, mode, should_raise",
-    (
-        (13, CompressWeightsMode.INT8_ASYM, False),
-        (13, CompressWeightsMode.INT8_SYM, False),
-        (13, CompressWeightsMode.INT4_SYM, True),
-        (13, CompressWeightsMode.INT4_ASYM, True),
-        (21, CompressWeightsMode.INT8_ASYM, False),
-        (21, CompressWeightsMode.INT8_SYM, False),
-        (21, CompressWeightsMode.INT4_SYM, False),
-        (21, CompressWeightsMode.INT4_ASYM, False),
-    ),
-)
-def test_raises_error_model_opset_versions(opset, mode, should_raise):
-    model = create_model(opset)
-    if should_raise:
-        with pytest.raises(nncf.ValidationError):
-            compress_weights(model, mode)
-    else:
-        compress_weights(model, mode)
+def test_matmulnbits():
+    np.random.seed(42)
+    model_opset21 = create_model()
+
+    np.random.seed(42)
+    model_opset19 = create_model(opset_version=19)
+
+    compressed_model_opset21 = compress_weights(model_opset21, mode=CompressWeightsMode.INT4_SYM, group_size=16)
+    compressed_model_opset19 = compress_weights(model_opset19, mode=CompressWeightsMode.INT4_SYM, group_size=16)
+
+    dummy_input = np.random.rand(100, 1280).astype(np.float32)
+
+    sess21 = InferenceSession(compressed_model_opset21.SerializeToString())
+    sess19 = InferenceSession(compressed_model_opset19.SerializeToString())
+
+    output21 = sess21.run(None, {"input": dummy_input})[0]
+    output19 = sess19.run(None, {"input": dummy_input})[0]
+
+    assert np.allclose(output21, output19, rtol=1e-5, atol=1e-6)
diff --git a/tests/onnx/test_onnx_helper.py b/tests/onnx/test_onnx_helper.py
index 565dda3268d..8f1bbed0770 100644
--- a/tests/onnx/test_onnx_helper.py
+++ b/tests/onnx/test_onnx_helper.py
@@ -11,11 +11,14 @@
 
 import tempfile
 
+import numpy as np
 import onnx
+from numpy.testing import assert_array_equal
 
 from nncf.onnx.graph.model_metadata import MetadataKey
 from nncf.onnx.graph.model_metadata import set_metadata
 from nncf.onnx.graph.onnx_helper import get_array_from_tensor
+from nncf.onnx.graph.onnx_helper import pack_int4_to_uint8
 from tests.onnx.models import build_matmul_model
 
 
@@ -50,3 +53,54 @@ def test_get_array_from_tensor_external_data():
             if x.name == "W":
                 tensor = x
         assert get_array_from_tensor(model, tensor).shape == (3, 2)
+
+
+def test_pack_int4_to_uint8_unsigned_even_block():
+    weight = np.array([[1, 2], [3, 4], [5, 6], [7, 8]], dtype=np.uint8)  # shape (4, 2)
+
+    packed = pack_int4_to_uint8(weight, block_size=4, signed=False)
+    expected = np.array(
+        [
+            [[(3 << 4) | 1, (7 << 4) | 5]],  # col 0: [1,3,5,7] -> [(3,1), (7,5)]
+            [[(4 << 4) | 2, (8 << 4) | 6]],  # col 1: [2,4,6,8] -> [(4,2), (8,6)]
+        ],
+        dtype=np.uint8,
+    )
+
+    assert packed.shape == (2, 1, 2)
+    assert_array_equal(packed, expected)
+
+
+def test_pack_int4_to_uint8_signed():
+    weight = np.array([[-8, -7], [-1, 0], [1, 2], [7, 7]], dtype=np.int8)
+
+    # Apply offset: signed=True -> +8 to each value
+    # Col 0: [-8, -1, 1, 7] -> [0, 7, 9, 15]
+    # Packed as [(7 << 4) | 0, (15 << 4) | 9] = [112, 249]
+    # Col 1: [-7, 0, 2, 7] -> [1, 8, 10, 15]
+    # Packed as: [(8 << 4) | 1, (15 << 4) | 10] = [129, 250]
+
+    packed = pack_int4_to_uint8(weight, block_size=4, signed=True)
+
+    expected = np.array([[[(7 << 4) | 0, (15 << 4) | 9]], [[(8 << 4) | 1, (15 << 4) | 10]]], dtype=np.uint8)
+
+    assert packed.shape == (2, 1, 2)
+    assert_array_equal(packed, expected)
+
+
+def test_pack_int4_to_uint8_multiple_blocks():
+    weight = np.array([[1], [2], [3], [4], [5], [6], [7], [8]], dtype=np.uint8)  # shape (8, 1)
+
+    packed = pack_int4_to_uint8(weight, block_size=4, signed=False)
+    expected = np.array(
+        [
+            [
+                [(2 << 4) | 1, (4 << 4) | 3],  # block 0
+                [(6 << 4) | 5, (8 << 4) | 7],  # block 1
+            ]
+        ],
+        dtype=np.uint8,
+    )
+
+    assert packed.shape == (1, 2, 2)
+    assert_array_equal(packed, expected)
diff --git a/tests/post_training/data/wc_reference_data.yaml b/tests/post_training/data/wc_reference_data.yaml
index 8824f966fe1..2dc1518a66f 100644
--- a/tests/post_training/data/wc_reference_data.yaml
+++ b/tests/post_training/data/wc_reference_data.yaml
@@ -2,6 +2,14 @@ tinyllama_data_free_backend_OV:
   metric_value: 0.73873
   num_int4: 114
   num_int8: 84
+tinyllama_data_free_backend_ONNX:
+  metric_value: 0.73779
+  num_int4: 264
+  num_int8: 84
+tinyllama_data_free_opset19_backend_ONNX:
+  metric_value: 0.73779
+  num_int4: 264
+  num_int8: 84
 tinyllama_data_aware_backend_OV:
   metric_value: 0.85767
   num_int4: 94
diff --git a/tests/post_training/data/wc_test_durations.json b/tests/post_training/data/wc_test_durations.json
index e6f76d30b9c..16959fa1b60 100644
--- a/tests/post_training/data/wc_test_durations.json
+++ b/tests/post_training/data/wc_test_durations.json
@@ -13,5 +13,7 @@
     "tests/post_training/test_quantize_conformance.py::test_weight_compression[tinyllama_NF4_scale_estimation_stateful_per_channel_backend_OV]": 193,
     "tests/post_training/test_quantize_conformance.py::test_weight_compression[tinyllama_scale_estimation_per_channel_backend_OV]": 251,
     "tests/post_training/test_quantize_conformance.py::test_weight_compression[tinyllama_data_free_awq_backend_OV]": 164,
-    "tests/post_training/test_quantize_conformance.py::test_weight_compression[tinyllama_data_free_awq_backend_TORCH]": 210
+    "tests/post_training/test_quantize_conformance.py::test_weight_compression[tinyllama_data_free_awq_backend_TORCH]": 210,
+    "tests/post_training/test_quantize_conformance.py::test_weight_compression[tinyllama_data_free_backend_ONNX]": 508,
+    "tests/post_training/test_quantize_conformance.py::test_weight_compression[tinyllama_data_free_opset19_backend_ONNX]": 512
 }
diff --git a/tests/post_training/model_scope.py b/tests/post_training/model_scope.py
index 9ed937bcefd..a35a47f65db 100644
--- a/tests/post_training/model_scope.py
+++ b/tests/post_training/model_scope.py
@@ -403,7 +403,20 @@
             "mode": CompressWeightsMode.INT4_SYM,
             "sensitivity_metric": SensitivityMetric.WEIGHT_QUANTIZATION_ERROR,
         },
-        "backends": [BackendType.OV],
+        "backends": [BackendType.OV, BackendType.ONNX],
+    },
+    {
+        "reported_name": "tinyllama_data_free_opset19",
+        "model_id": "tinyllama/tinyllama-1.1b-step-50k-105b",
+        "pipeline_cls": LMWeightCompression,
+        "compression_params": {
+            "group_size": 64,
+            "ratio": 0.8,
+            "mode": CompressWeightsMode.INT4_SYM,
+            "sensitivity_metric": SensitivityMetric.WEIGHT_QUANTIZATION_ERROR,
+        },
+        "params": {"opset": 19},
+        "backends": [BackendType.ONNX],
     },
     {
         "reported_name": "tinyllama_data_aware",
diff --git a/tests/post_training/pipelines/lm_weight_compression.py b/tests/post_training/pipelines/lm_weight_compression.py
index 1ba3b68149b..6f441f5b8ec 100644
--- a/tests/post_training/pipelines/lm_weight_compression.py
+++ b/tests/post_training/pipelines/lm_weight_compression.py
@@ -18,12 +18,15 @@
 from typing import Optional
 
 import numpy as np
+import onnx
 import openvino as ov
 import torch
 from datasets import load_dataset
 from memory_profiler import memory_usage
+from optimum.exporters.onnx import main_export
 from optimum.exporters.openvino.convert import export_from_model
 from optimum.intel.openvino import OVModelForCausalLM
+from optimum.onnxruntime import ORTModelForCausalLM
 from transformers import AutoModelForCausalLM
 from transformers import AutoTokenizer
 from whowhatbench import Evaluator
@@ -93,6 +96,7 @@ class LMWeightCompression(BaseTestPipeline):
     """Pipeline for casual language models from Hugging Face repository"""
 
     OV_MODEL_NAME = "openvino_model.xml"
+    ONNX_MODEL_NAME = "model.onnx"
 
     def __init__(
         self,
@@ -157,6 +161,25 @@ def prepare_model(self) -> None:
                     self.fp32_model_dir, trust_remote_code=True, load_in_8bit=False, compile=False, stateful=is_stateful
                 )
             self.model = self.model_hf.model
+        elif self.backend == BackendType.ONNX:
+            if is_stateful:
+                msg = f"is_stateful={is_stateful} is not supported for {self.backend.value} backend."
+                raise RuntimeError(msg)
+
+            if not (self.fp32_model_dir / self.ONNX_MODEL_NAME).exists():
+                opset_version = self.params.get("opset", None)
+                if opset_version:
+                    main_export(self.model_id, self.fp32_model_dir, opset=opset_version)
+                    self.model_hf = ORTModelForCausalLM.from_pretrained(self.fp32_model_dir, trust_remote_code=True)
+                else:
+                    self.model_hf = ORTModelForCausalLM.from_pretrained(self.model_id, export=True)
+                    self.model_hf.save_pretrained(self.fp32_model_dir)
+
+            else:
+                self.model_hf = ORTModelForCausalLM.from_pretrained(self.fp32_model_dir, trust_remote_code=True)
+
+            self.model = onnx.load(self.fp32_model_dir / self.ONNX_MODEL_NAME, load_external_data=False)
+
         else:
             msg = f"backend={self.backend.value} is not supported."
             raise RuntimeError(msg)
@@ -299,6 +322,14 @@ def save_compressed_model(self) -> None:
                     msg = "Openvino Model Files Not Found!"
                     raise FileNotFoundError(msg)
                 self.path_compressed_ir = cached_ov_model_files[0]
+        elif self.backend == BackendType.ONNX:
+            self.model_hf._save_config(self.output_model_dir)
+            onnx.save(
+                self.compressed_model,
+                str(self.output_model_dir / self.ONNX_MODEL_NAME),
+                save_as_external_data=True,
+                location="model.onnx_data",
+            )
 
     def run_bench(self) -> None:
         pass
@@ -322,11 +353,25 @@ def _compress(self):
         """
         Actual call of weight compression
         """
-        self.compressed_model = nncf.compress_weights(
-            self.model,
-            dataset=self.calibration_dataset,
-            **self.compression_params,
-        )
+        if self.backend == BackendType.ONNX:
+            from nncf.onnx.quantization.backend_parameters import BackendParameters
+
+            advanced_params = self.compression_params.setdefault(
+                "advanced_parameters", nncf.AdvancedCompressionParameters()
+            )
+            advanced_params.backend_params = {BackendParameters.EXTERNAL_DATA_DIR: str(self.fp32_model_dir.absolute())}
+
+            self.compressed_model = nncf.compress_weights(
+                self.model,
+                dataset=None,  # ONNX backend supports only data free compression
+                **self.compression_params,
+            )
+        else:
+            self.compressed_model = nncf.compress_weights(
+                self.model,
+                dataset=self.calibration_dataset,
+                **self.compression_params,
+            )
 
     def _validate(self) -> None:
         is_stateful = self.params.get("is_stateful", False)
@@ -359,7 +404,11 @@ def _validate(self) -> None:
             )
 
         compressed_model_hf = self.model_hf
-        if self.backend != BackendType.FP32 and self.backend != BackendType.FX_TORCH:
+        if (
+            self.backend != BackendType.FP32
+            and self.backend != BackendType.FX_TORCH
+            and self.backend != BackendType.ONNX
+        ):
             compressed_model_hf = OVModelForCausalLM.from_pretrained(
                 self.output_model_dir,
                 trust_remote_code=True,
@@ -370,6 +419,19 @@ def _validate(self) -> None:
             )
         if self.backend == BackendType.FX_TORCH:
             compressed_model_hf = FXAutoModelForCausalLM(self.model, self.model_config)
+
+        if self.backend == BackendType.ONNX:
+            compressed_model_hf = OVModelForCausalLM.from_pretrained(
+                self.output_model_dir,
+                trust_remote_code=True,
+                load_in_8bit=False,
+                compile=False,
+                stateful=False,
+                ov_config={"DYNAMIC_QUANTIZATION_GROUP_SIZE": "0", "KV_CACHE_PRECISION": "f16"},
+                export=False,
+                from_onnx=True,
+            )
+
         print("Evaluation of the target model")
         _, all_metrics = evaluator.score(compressed_model_hf)
         similarity = all_metrics["similarity"][0]
@@ -377,8 +439,22 @@ def _validate(self) -> None:
         self.run_info.metric_value = round(similarity, 5)
 
     def get_num_compressed(self) -> None:
-        ie = ov.Core()
-        model = ie.read_model(model=self.path_compressed_ir)
+        if self.backend == BackendType.ONNX:
+            compressed_model_hf = OVModelForCausalLM.from_pretrained(
+                self.output_model_dir,
+                trust_remote_code=True,
+                load_in_8bit=False,
+                compile=False,
+                stateful=False,
+                ov_config={"DYNAMIC_QUANTIZATION_GROUP_SIZE": "0", "KV_CACHE_PRECISION": "f16"},
+                export=False,
+                from_onnx=True,
+            )
+            model = compressed_model_hf.model
+        else:
+            ie = ov.Core()
+            model = ie.read_model(model=self.path_compressed_ir)
+
         _, num_int4, num_int8 = get_num_fq_int4_int8(model)
 
         self.run_info.num_compress_nodes.num_int8 = num_int8


Reviews:
user: login: anzr299
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-19T14:28:53Z
user: login: alexsu52
body: Please add an unit test for MatMulNBits + pack_int4_to_uint8
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T05:18:01Z
user: login: andrey-churkin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T10:41:24Z
user: login: andrey-churkin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T10:45:39Z
user: login: AlexanderDokuchaev
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-23T16:00:00Z
user: login: alexsu52
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-25T08:16:49Z
user: login: andrey-churkin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-26T08:40:11Z

Review comments:
diff_hunk: @@ -481,3 +481,14 @@ def remove_initializer(initializer_name: str, model: onnx.ModelProto) -> None:
     original_initializer = next((init for init in model.graph.initializer if init.name == initializer_name), None)
     if original_initializer is not None:
         model.graph.initializer.remove(original_initializer)
+
+
+def remove_node(node_name: str, model: onnx.ModelProto) -> None:
+    """
+    Removes the node from the ONNX model.
path: nncf/onnx/graph/model_transformer.py
user: login: AlexanderDokuchaev
body: ```suggestion
    Removes the node from the ONNX model.
    
```
created_at: 2025-05-21T09:20:00Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,179 @@
+,prompts,answers,language
path: tests/post_training/data/wwb_ref_answers/tinyllama__tinyllama-1.1b-step-50k-105b/ref_qa_onnx.csv
user: login: andrey-churkin
body: @alexsu52 The `req_qa_onnx.csv` file was obtained using the following "gold" model:
```python
model_gold = ORTModelForCausalLM.from_pretrained(
    self.fp32_model_dir,
    trust_remote_code=True,
    provider="OpenVINOExecutionProvider"
)
```
It is identical to `ref_qa.csv`, which was generated using the model below:

```python
model_gold = OVModelForCausalLM.from_pretrained(
    self.fp32_model_dir,
    trust_remote_code=True,
    load_in_8bit=False,
    compile=False,
    stateful=is_stateful,
    ov_config={"KV_CACHE_PRECISION": "f16"},
)
````
So, if you don't mind, I suggest using only the `ref_qa.csv` file.
created_at: 2025-05-22T10:41:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -157,6 +161,25 @@ def prepare_model(self) -> None:
                     self.fp32_model_dir, trust_remote_code=True, load_in_8bit=False, compile=False, stateful=is_stateful
                 )
             self.model = self.model_hf.model
+        elif self.backend == BackendType.ONNX:
+            if is_stateful:
+                msg = f"is_stateful={is_stateful} is not supported for {self.backend.value} backend."
+                raise RuntimeError(msg)
+
+            if not (self.fp32_model_dir / self.ONNX_MODEL_NAME).exists():
+                opset_version = self.params.get("opset", None)
+                if opset_version:
+                    main_export(self.model_id, self.fp32_model_dir, opset=opset_version)
+                    self.model_hf = ORTModelForCausalLM.from_pretrained(self.fp32_model_dir, trust_remote_code=True)
+                else:
+                    self.model_hf = ORTModelForCausalLM.from_pretrained(self.model_id, export=True)
+                    self.model_hf.save_pretrained(self.fp32_model_dir)
path: tests/post_training/pipelines/lm_weight_compression.py
user: login: andrey-churkin
body: We can't specify the opset version with `ORTModelForCausalLM.from_pretrained()`. To obtain a model with an opset version lower than 21, we should use the `main_export()` method (or a similar approach).
created_at: 2025-05-22T10:45:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -328,3 +368,94 @@ def _add_dequantize_linear_layer(
         # Update the node mapping
         self.name_to_node_map[dequantize_node.name] = dequantize_node
         return model
+
+    def _replace_matmul_with_matmulnbits(
+        self,
+        model: onnx.ModelProto,
+        weight_compression_parameters: WeightCompressionParameters,
+        quantized_weights: np.ndarray,
+        orig_weight,
+        scale: np.ndarray,
+        zero_point: Optional[np.ndarray],
+        axis: int,
+        block_size: int,
+        weight_name: str,
+        weight_dtype: onnx.TensorProto.DataType,
+        scale_dtype: onnx.TensorProto.DataType = onnx.TensorProto.FLOAT,
+    ):
+        if weight_dtype == onnx.TensorProto.INT4:
path: nncf/quantization/algorithms/weight_compression/onnx_backend.py
user: login: AlexanderDokuchaev
body: Please add else block with rising error 
created_at: 2025-05-23T15:58:44Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,179 @@
+,prompts,answers,language
path: tests/post_training/data/wwb_ref_answers/tinyllama__tinyllama-1.1b-step-50k-105b/ref_qa_onnx.csv
user: login: alexsu52
body: ok
created_at: 2025-05-25T08:16:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -328,3 +368,94 @@ def _add_dequantize_linear_layer(
         # Update the node mapping
         self.name_to_node_map[dequantize_node.name] = dequantize_node
         return model
+
+    def _replace_matmul_with_matmulnbits(
+        self,
+        model: onnx.ModelProto,
+        weight_compression_parameters: WeightCompressionParameters,
+        quantized_weights: np.ndarray,
+        orig_weight,
+        scale: np.ndarray,
+        zero_point: Optional[np.ndarray],
+        axis: int,
+        block_size: int,
+        weight_name: str,
+        weight_dtype: onnx.TensorProto.DataType,
+        scale_dtype: onnx.TensorProto.DataType = onnx.TensorProto.FLOAT,
+    ):
+        if weight_dtype == onnx.TensorProto.INT4:
path: nncf/quantization/algorithms/weight_compression/onnx_backend.py
user: login: andrey-churkin
body: I guess we should raise an error when `weight_type` is not in [`onnx.TensorProto.INT4`, `onnx.TensorProto.UINT4`]. We just shouldn't pack to uint8 in that case.
created_at: 2025-05-26T08:40:11Z
author_association: CONTRIBUTOR

PR comments:
user: login: andrey-churkin
created_at: 2025-05-22T16:11:56Z
body: WC run: https://github.com/openvinotoolkit/nncf/actions/runs/15191579763
author_association: CONTRIBUTOR
user: login: MaximProshin
created_at: 2025-05-26T09:14:08Z
body: To be merged after https://github.com/openvinotoolkit/nncf/pull/3510 
author_association: COLLABORATOR
user: login: andrey-churkin
created_at: 2025-05-26T10:54:38Z
body: WC (Passed): https://github.com/openvinotoolkit/nncf/actions/runs/15251374658
author_association: CONTRIBUTOR