Repository: onnx/onnx

Issue:
INT4 TensorProto byte size is 5x larger than expected with negative values
# Bug Report

### Is the issue related to model conversion?
<!-- If the ONNX checker reports issues with this model then this is most probably related to the converter used to convert the original framework model to ONNX. Please create this bug in the appropriate converter's GitHub repo (pytorch, tensorflow-onnx, sklearn-onnx, keras-onnx, onnxmltools) to get the best help. -->
No

### Describe the bug
<!-- Please describe the bug clearly and concisely -->
An int4 tensor created with `onnx.helper.make_tensor()` is 5x larger than expected if the data values are **negative**. This only applies to tensors that use the `int32_data` field (not `raw_data`).

This is happening due to the following reasons:
- `onnx.helper.make_tensor()` is packing two INT4 values into `int32_data` as a single `np.int8` value. I believe this is incorrect. Should always pack two INT4 values as a single **`np.uint8`** value for both INT4 and UINT4.
- protobuf encodes a negative value using 10 bytes (instead of 2 bytes for a positive value in the range [0-255]).

### System information
<!--
- OS Platform and Distribution (*e.g. Linux Ubuntu 20.04*):
- ONNX version (*e.g. 1.13*):
- Python version:
- GCC/Compiler version (if compiling from source):
- CMake version:
- Protobuf version:
- Visual Studio version (if applicable):-->


### Reproduction instructions
Here's a python script to reproduce the issue. It creates an int4 tensor with all negative values. The byte size is 5x larger.

```python
# int4_size.py
import math
import numpy as np
import onnx

if __name__ == "__main__":
    data = np.array([-4] * 4096, dtype=np.int8) # 4096 negative values
    shape = (4096,)
    
    tensor_x = onnx.helper.make_tensor("x", onnx.TensorProto.INT4, shape, data)
    print(f"{len(tensor_x.int32_data)=}")
    print(f"{hex(tensor_x.int32_data[0])=}") # Should print '0xcc', but get '-0x34'!
    print(f"{tensor_x.ByteSize()=}") # Get 20,492 bytes! Huge! Expect ~4096 due to protobuf varint encoding.

    # Protobuf uses variable-width encoding for each element in int32_data
    # A uint8 value is encoded in 2 bytes. Ex: 0xcc is encoded as 0xcc10
    # A negative int8 value is encoded in 10 bytes. Ex: 0xcc (also -0x34) is encoded as 0xffffffffffffffcc
    # Expected ByteSize() = (4096/2) * 2 + other_fields_size = ~4096 (NOT 20k)

    num_elems = np.prod(shape) # 4096 int4 elements
    expected_size = math.ceil(num_elems / 2.0) # 2048 bytes are expected
    actual_size = len(bytes(tensor_x.int32_data)) # Expect 2048, but get exception "ValueError: bytes must be in range(0, 256)
                                                  # Each element in int32_data must be an opaque uint8 with bits[31:8] being zero
                                                  # so that protobuf can serialize the element as 1 byte tag + 1 byte data.
    print(f"{expected_size=} bytes, tensor_x.int32_data bytes = {actual_size} bytes")
```

The above script generates the following output. Note that tensor.ByteSize() is 20,492 bytes for an int4 tensor with 4096 4bit elements.
```shell
$ python int4_size.py

len(tensor_x.int32_data)=2048
hex(tensor_x.int32_data[0])='-0x34'
tensor_x.ByteSize()=20492
Traceback (most recent call last):
  File "int4_size.py", line 18, in <module>
    actual_size = len(bytes(tensor_x.int32_data)) # Expect 2048, but get exception "ValueError: bytes must be in range(0, 256)
ValueError: bytes must be in range(0, 256)
```
### Expected behavior
<!-- A clear and concise description of what you expected to happen. -->

The above script should generate the following output. Note that the tensor.ByteSize() is ~4096 bytes, which is expected due to how protobuf uses 2 bytes to encode a single byte. Also, calling `len(bytes(tensor_x.int32_data))` returns the expected 2048 bytes.

```shell
$ python int4_size.py

len(tensor_x.int32_data)=2048
hex(tensor_x.int32_data[0])='0xcc'
tensor_x.ByteSize()=4107
expected_size=2048 bytes, tensor_x.int32_data bytes = 2048 bytes
```
See https://github.com/onnx/onnx/pull/6161
### Notes
<!-- Any additional information -->



Discussion:

Resolving patch:
diff --git a/onnx/helper.py b/onnx/helper.py
--- a/onnx/helper.py
+++ b/onnx/helper.py
@@ -774,12 +774,13 @@ def make_tensor(
             TensorProto.INT4,
         ):
             signed = data_type == TensorProto.INT4
-            vals = (
-                pack_float32_to_4bit(vals, signed=signed)
-                .astype(np_dtype)
-                .flatten()
-                .tolist()
-            )
+
+            # Two packed 4-bit values must be represented as a single uint8 value.
+            # Therefore, pack_float32_to_4bit() sets the dtype of the output vals
+            # to uint8 regardless of the value of 'signed'. Using int8 would cause
+            # the size of int4 tensors to increase ~5x if the tensor contains negative values (due to
+            # the way negative values are serialized by protobuf).
+            vals = pack_float32_to_4bit(vals, signed=signed).flatten().tolist()
         elif data_type == TensorProto.BOOL:
             vals = np.array(vals).astype(int)
         elif data_type == TensorProto.STRING:
diff --git a/onnx/reference/op_run.py b/onnx/reference/op_run.py
--- a/onnx/reference/op_run.py
+++ b/onnx/reference/op_run.py
@@ -164,7 +164,9 @@ def to_array_extended(tensor: TensorProto) -> np.ndarray:
         m = {TensorProto.INT4: int4, TensorProto.UINT4: uint4}
         dtype = m[elem_type]  # type: ignore[index]
         signed = elem_type == TensorProto.INT4
-        y = np.empty(len(data), dtype=dtype).ravel()
+        # 2 packed int4 elements must be represented as a single uint8 value.
+        # Therefore, y is np.uint8 (not the dtype to which the int4 maps)
+        y = np.empty(len(data), dtype=np.uint8).ravel()  # type: ignore[assignment]
         for i, d in enumerate(data):
             y[i] = d
 


Test patch:
diff --git a/onnx/test/helper_test.py b/onnx/test/helper_test.py
--- a/onnx/test/helper_test.py
+++ b/onnx/test/helper_test.py
@@ -4,6 +4,7 @@
 from __future__ import annotations
 
 import itertools
+import math
 import random
 import struct
 import unittest
@@ -656,9 +657,37 @@ def test_make_4bit_tensor(self, dtype, dims) -> None:
             type_range[dtype][0], high=type_range[dtype][1] + 1, size=dims
         )
         y = helper.make_tensor("y", dtype, data.shape, data)
+
+        # Check the expected size of int32_data in bytes
+        expected_data_size = math.ceil(np.prod(data.shape) / 2.0)
+        actual_data_size = len(bytes(y.int32_data))
+        np.testing.assert_equal(actual_data_size, expected_data_size)
+
+        # Check the expected data values.
         ynp = to_array_extended(y)
         np.testing.assert_equal(data, ynp)
 
+    @parameterized.parameterized.expand(
+        itertools.product(
+            ((5, 4, 6), (4, 6, 5), (3, 3), (1,), (2**10,)),
+        )
+    )
+    @unittest.skipIf(
+        version_utils.numpy_older_than("1.22.0"),
+        "The test requires numpy 1.22.0 or later",
+    )
+    def test_4bit_tensor_size(self, dims) -> None:
+        # A bug caused negative int4 values to inflate tensor size.
+        # So, test negative values here.
+        num_elems = np.prod(dims)
+        data = np.array([-4] * num_elems, dtype=np.int8).reshape(dims)
+        y = helper.make_tensor("y", TensorProto.INT4, data.shape, data)
+
+        # Check the expected size of int32_data in bytes
+        expected_data_size = math.ceil(num_elems / 2.0)
+        actual_data_size = len(bytes(y.int32_data))
+        np.testing.assert_equal(actual_data_size, expected_data_size)
+
     @parameterized.parameterized.expand(
         itertools.product(
             (TensorProto.UINT4, TensorProto.INT4), ((5, 4, 6), (4, 6, 5), (3, 3), (1,))