Multiplying two uint8 tensors without overflow

lewisc: <p>I want to multiply two uint8, for example</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.randint(low=0,high=255, size=(5,), dtype=torch.uint8)
&gt;&gt;&gt; b = torch.randint(low=0,high=255, size=(5,), dtype=torch.uint8)
&gt;&gt;&gt; a
tensor([212, 181,   3,  89, 147], dtype=torch.uint8)
&gt;&gt;&gt; b
tensor([220, 207, 104, 228, 172], dtype=torch.uint8)
&gt;&gt;&gt; a * b
tensor([ 48,  91,  56,  68, 196], dtype=torch.uint8)
</code></pre>
<p>However, the return type by default is also uint8, thus it causes overflow issue. The result for the first element in the array is 212 * 220 % 256 = 48, but I want 212 * 220 = 46640. I know it is impossible to hold the result in dtype uint8, but I cannot find any way to store the result in a tensor with high bits. I have also tried the following:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; c = torch.empty(5, dtype=torch.int16)
&gt;&gt;&gt; torch.mul(a, b, out=c)
tensor([ 48,  91,  56,  68, 196], dtype=torch.int16)
</code></pre>
<p>Though the results is stored in an int16 tensor now, it is still truncated.</p>
<p>I know I can first convert the uint8 tensor to int16, then do the multiplication, but that would cause extra overhead and I want to avoid that. Therefore I am wondering if it is possible to store the result in a higher bit tensor without truncating. Thanks</p>

Zafar: <p>We probably can change the quantization to always set to max/min when overflowing.</p>

Zafar: <p>To clarify, the current question is not about quantization as defined by the torch.ao, but rather the low-precision dtypes. However, I think the quantized dtypes should be aware of the current behavior</p>

lewisc: <aside class="quote no-group" data-username="Zafar" data-post="3" data-topic="155181">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zafar/48/11532_2.png" class="avatar"> Zafar:</div>
<blockquote>
<p>not</p>
</blockquote>
</aside>
<p>Hi Zafar,</p>
<p>I agree this question is not about quantization, but I cannot find a subject that’s more appropriate. I thought this question should be frequently dealt when doing int8 arithmetics for quantization. If you have any suggestions about the subject, I am glad to change it.</p>
<p>I am asking this question from a pure math angle. Is there any way to get the results without overflow? For example, is there any python or cpp API I can use?</p>
<p>Thanks</p>

Zafar: <aside class="quote no-group quote-modified" data-username="lewisc" data-post="1" data-topic="155181">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lewisc/48/40567_2.png" class="avatar"> lewisc:</div>
<blockquote>
<p>I am asking this question from a pure math angle. Is there any way to get the results without overflow?</p>
</blockquote>
</aside>
<p>IMHO, from the mathematical PoV, this is expected behavior. The integer arithmetic follows the logic of the modular arithmetic, and hence the behavior is as expected. You can notice the same behavior across different frameworks.</p>
<p>For example, in numpy you would get identical results:</p>
<pre><code class="lang-auto">a_list = [212, 181,   3,  89, 147]
b_list = [220, 207, 104, 228, 172]

a_np = np.array(a_list, dtype=np.uint8)
b_np = np.array(b_list, dtype=np.uint8)

c_np = a_np * b_np
print(f'c_np={c_np}, c_np.dtype={c_np.dtype}')
# c_np=[ 48  91  56  68 196], c_np.dtype=uint8

c_np_out = np.zeros(shape=a_np.shape, dtype=np.uint16)
c_np_out = np.multiply(a_np, b_np, out=c_np_out)
print(f'c_np_out={c_np_out}, c_np_out.dtype={c_np_out.dtype}')
# c_np_out=[ 48  91  56  68 196], c_np_out.dtype=uint16
</code></pre>
<hr>
<aside class="quote no-group quote-modified" data-username="lewisc" data-post="1" data-topic="155181">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lewisc/48/40567_2.png" class="avatar"> lewisc:</div>
<blockquote>
<p>For example, is there any python or cpp API I can use?</p>
</blockquote>
</aside>
<p>As far as the options go, you have several, ranging from simple boundary checks, promotions, and ending with torch customizations.</p>
<p><em>Not describing the boundary checks, as those usually application specific</em></p>
<h2><a name="option-1-manual-type-promotion-1" class="anchor" href="#option-1-manual-type-promotion-1"></a>Option 1. Manual type promotion</h2>
<pre><code class="lang-auto">a_list = [212, 181,   3,  89, 147]
b_list = [220, 207, 104, 228, 172]

a_torch = torch.tensor(a_list, dtype=torch.uint8)
b_torch = torch.tensor(b_list, dtype=torch.uint8)

c_torch = a_torch * b_torch.to(torch.int32)
print(f'c_torch={c_torch}')

c_torch_out = torch.zeros(size=a_torch.shape, dtype=torch.int16)
c_torch_out = torch.mul(a_torch, b_torch, out=c_torch_out)
print(f'c_torch={c_torch_out}')
</code></pre>
<h2><a name="option-2-torch-customizations-2" class="anchor" href="#option-2-torch-customizations-2"></a>Option 2. Torch customizations</h2>
<p>This one is following the documentation here: <a href="https://pytorch.org/docs/stable/notes/extending.html" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 2.1 documentation</a></p>
<pre><code class="lang-auto"># Create a bounded tensor
class BoundedIntTensor:
    HANDLED_FUNCTIONS = {}
    def __init__(self, data, lo=0, hi=255, **kwargs):
        self._t = torch.as_tensor(data, **kwargs)
        self.lo = lo
        self.hi = hi

    def __repr__(self):
        return f"BoundedIntTensor ({self.lo}, {self.hi})\n\tdata: {self._t}"

    @classmethod
    def __torch_function__(cls, func, types, args=(), kwargs=None):
        if kwargs is None:
            kwargs = {}
        if func not in cls.HANDLED_FUNCTIONS or not all(
            issubclass(t, (torch.Tensor, BoundedIntTensor))
            for t in types
        ):
            args = [a.tensor() if hasattr(a, 'tensor') else a for a in args]
            return func(*args, **kwargs)
        return cls.HANDLED_FUNCTIONS[func](*args, **kwargs)
    
# Implement custom multiplication logic
import functools
def implements(tensor_type, torch_function):
    """Register a torch function override for ScalarTensor"""
    @functools.wraps(torch_function)
    def decorator(func):
        tensor_type.HANDLED_FUNCTIONS[torch_function] = func
        return func
    return decorator

@implements(BoundedIntTensor, torch.mul)
def mul(self, other):
    result = self._t.to(torch.int32) * other._t
    result[result &gt; self.hi] = self.hi
    result[result &lt; self.lo] = self.lo
    return BoundedIntTensor(result.to(self._t.dtype), lo=self.lo, hi=self.hi)

# Check the result
a_list = [212, 12, 3,  89, 147]
b_list = [220, 10, 3, 228, 172]

a = BoundedIntTensor(a_list, lo=0, hi=123)
b = BoundedIntTensor(b_list, lo=0, hi=123)

print(torch.mul(a, b))
# BoundedIntTensor (0, 123)
# 	data: tensor([123, 120,   9, 123, 123])
</code></pre>

lewisc: <p>Sorry for the confusion, I understand this is an expected behavior. I was just hoping to find an alternative way to bypass this overflow issue and get the mathematically correct answer.</p>
<p>So grateful for your detailed reply! If I understand correctly, the underlying arithmetics for the both options are multiplications of int32. However, as I mentioned earlier, the reason I wanted to this is because I wanted to take advantage of the speedup of uint8 multiplication. If I do the multiplication after promoting the uint8 tensor to int32, I am not sure how efficient it will be.</p>
<p>One reason I posted this issue at the quantization channel is because this issue should be universal in quantization. For example, how is the quantized linear layer is computed?</p>
<p>If we don’t consider zero point, scaling factor, and bias for now, the linear layer boils down to multiplication of two int8 tensor (weight and input). To get the correct answer, the result of this multiplication cannot be overflow. I thought the multiplication is done in int8 and the result is stored in a tensor with higher bits and then converted back into int8. I am curious how is this step efficiently implemented to get the ideal speedup?</p>

Zafar: <blockquote>
<p>the reason I wanted to this is because I wanted to take advantage of the speedup of uint8 multiplication</p>
</blockquote>
<p>IMHO, To take advantage of uint8 arithmetic, you would have to make some assumptions, such as input/output ranges. Otherwise, there is no way knowing if multiplication is overflowing.</p>
<blockquote>
<p>I am curious how is this step efficiently implemented to get the ideal speedup?</p>
</blockquote>
<p>Quantization doesn’t make assumptions, and forces the user to provide the output scale and zero_point (<a href="https://github.com/pytorch/pytorch/blob/2a09e9516942967416c2a5983960ccba8bcf55cd/aten/src/ATen/native/quantized/cpu/qmul.cpp#L108" rel="noopener nofollow ugc">see here</a>). Here is an example:</p>
<pre><code class="lang-auto">a_list = [212, 181,   3,  89, 147]
b_list = [220, 207, 3, 228, 172]

a_torch = torch.tensor(a_list, dtype=torch.uint8)
b_torch = torch.tensor(b_list, dtype=torch.uint8)

qa_torch = torch.quantize_per_tensor(a_torch.to(torch.float), scale=1.0, zero_point=0, dtype=torch.quint8)
qb_torch = torch.quantize_per_tensor(b_torch.to(torch.float), scale=1.0, zero_point=0, dtype=torch.quint8)

print(f'- qa_torch={qa_torch}') 
print(f'- qa_torch.int_repr={qa_torch.int_repr()}')
print(f'- qb_torch={qb_torch}')
print(f'- qb_torch.int_repr={qb_torch.int_repr()}')
# - qa_torch=tensor([212., 181.,   3.,  89., 147.], size=(5,), dtype=torch.quint8,
#        quantization_scheme=torch.per_tensor_affine, scale=1.0, zero_point=0)
# - qa_torch.int_repr=tensor([212, 181,   3,  89, 147], dtype=torch.uint8)
# - qb_torch=tensor([220., 207.,   3., 228., 172.], size=(5,), dtype=torch.quint8,
#        quantization_scheme=torch.per_tensor_affine, scale=1.0, zero_point=0)
# - qb_torch.int_repr=tensor([220, 207,   3, 228, 172], dtype=torch.uint8)

qc_torch = torch.ops.quantized.mul(qa_torch, qb_torch, scale=1.0, zero_point=0)
print(f'- qc_torch={qc_torch}')
print(f'- qc_torch.int_repr={qc_torch.int_repr()}')
# - qc_torch=tensor([255., 255.,   9., 255., 255.], size=(5,), dtype=torch.quint8,
#        quantization_scheme=torch.per_tensor_affine, scale=1.0, zero_point=0)
# - qc_torch.int_repr=tensor([255, 255,   9, 255, 255], dtype=torch.uint8)
</code></pre>

ParGG: <blockquote>
<p>One reason I posted this issue at the quantization channel is because this issue should be universal in quantization. For example, how is the quantized linear layer is computed?</p>
</blockquote>
<blockquote>
<p>If we don’t consider zero point, scaling factor, and bias for now, the linear layer boils down to multiplication of two int8 tensor (weight and input). To get the correct answer, the result of this multiplication cannot be overflow. I thought the multiplication is done in int8 and the result is stored in a tensor with higher bits and then converted back into int8. I am curious how is this step efficiently implemented to get the ideal speedup?</p>
</blockquote>
<p>Here is how the operation is performed: <a href="https://github.com/pytorch/FBGEMM/blob/9d7c48a65419d0350f9e9e72f31e05bfe37e85a4/include/fbgemm/OutputProcessing-inl.h#L59" rel="noopener nofollow ugc">https://github.com/pytorch/FBGEMM/blob/9d7c48a65419d0350f9e9e72f31e05bfe37e85a4/include/fbgemm/OutputProcessing-inl.h#L59</a></p>
<p>Basically everything is performed in int32; then, when considering the scale, it gets converted to float, and finally it gets rounded to long right before clipping.</p>

lewisc: <p>Thanks, Zafar and ParGG!</p>
<p>I think my previous understanding about quantization is incorrect. zero point and scaling factor are also indivisible part of quantization.</p>
<p>As ParGG said, if all the int8 arithmetics in FBGEMM is also performed with int32, I should not worry about the efficiency issue of type promotion.</p>
<p>This is the first time I posted on PyTorch forum, and I find you guys are very helpful! I really appreciate your help</p>