Bug: Support Int128 type
Status: Open | Priority: Major | Resolution: 
Created: 16/Aug/22 06:14
Affects Version/s: 3.4.0

Spark SQL today supports the Decimal data type. The implementation of Spark Decimal holds a BigDecimal or Long value. Spark Decimal provides some operators like +, -, *, /, % and so on. These operators rely heavily on the computational power of BigDecimal or Long itself. For ease of understanding, take the + as an example. The implementation shows below.

{code:java}
  def + (that: Decimal): Decimal = {
    if (decimalVal.eq(null) && that.decimalVal.eq(null) && scale == that.scale) {
      Decimal(longVal + that.longVal, Math.max(precision, that.precision) + 1, scale)
    } else {
      Decimal(toBigDecimal.bigDecimal.add(that.toBigDecimal.bigDecimal))
    }
  }
{code}

We can see the + of Long will be called if Spark Decimal holds a Long value. Otherwise,  the add of BigDecimal will be called if Spark Decimal holds a BigDecimal value. The other operators of Spark Decimal adopt the similar way. Furthermore, the code shown above calls Decimal.apply to construct a new instance of Spark Decimal.  As we know, the add operator of BigDecimal constructed a new instance of BigDecimal. So, if we call the + operator of Spark Decimal who holds a Long value, Spark will construct a new instance of Decimal. Otherwise, Spark will construct a new instance of BigDecimal and a new instance of Decimal.
Through rough analysis, we know:
1. The computational power of Spark Decimal may depend on BigDecimal.
2. The calculation operators of Spark Decimal create a lot of new instances of Decimal and may create a lot of new instances of BigDecimal.
If a large table has a field called 'colA whose type is Spark Decimal, the execution of SUM('colA) will involve the creation of a large number of Spark Decimal instances and BigDecimal instances. These Spark Decimal instances and BigDecimal instances will lead to garbage collection frequently.
Int128 is a high-performance data type about 2X~10X more efficient than Spark Decimal for typical operations. It uses a finite (128 bit) precision and can handle up to decimal(38, X). The implementation of Int128 just uses two Long values to represent the high and low bits of 128 bits respectively. Int128 is lighter than Spark Decimal, reduces the cost of new() and garbage collection.

h3. Milestone 1 – Spark Decimal equivalency ( The new type Int128 meets or exceeds all function of the existing SQL Decimal):
* Add a new DataType implementation for Int128.
* Support Int128 in Dataset/UDF.
* Int128 literals
* Int128 arithmetic(e.g. Int128 + Int128, Int128 - Decimal)
* Decimal or Math functions/operators: POWER, LOG, Round, etc
* Cast to and from Int128, cast String/Decimal to Int128, cast Int128 to string (pretty printing)/Decimal, with the * * SQL syntax to specify the types
* Support sorting Int128.

h3. Milestone 2 – Persistence:
 * Ability to create tables of type Int128
 * Ability to write to common file formats such as Parquet and JSON.
 * INSERT, SELECT, UPDATE, MERGE
 * Discovery

h3. Milestone 3 – Client support
 * JDBC support
 * Hive Thrift server

h3. Milestone 4 – PySpark and Spark R integration
 * Python UDF can take and return Int128
 * DataFrame support