dynamobi/dynamobi — Issue #300: [LDB-21] Neither Java nor Fennel calc handles arithmetic overflow
Author: dynamobi-build
Created: 2011-12-30 02:28:11+00:00
State: open
Labels: LDB

[reporter="jpham", created="Fri, 3 Feb 2006 15:29:21 -0500 (GMT-05:00)"]
For example, take the following integer addition

0: jdbc:farrago:> values 2100000000 + 2100000000;

+------------+

| EXPR$0 |

+------------+

| -94967296 |

+------------+

Make the numbers a little larger (so they are bigint) and things work

0: jdbc:farrago:> values 3100000000 + 3100000000;

+-------------+

| EXPR$0 |

+-------------+

| 6200000000 |

+-------------+

But you still have problems

0: jdbc:farrago:> values 8876543210987654321 + 1000000000000000000;

+-----------------------+

|EXPR$0 |

+-----------------------+

| -8570200862721897295 |

+-----------------------+

I think the Fennel representation depends on the Boost type / arithmetic libraries while the generated Java code doesn't work because it checks the result too late.

On the positive side, both Java and Fennel calculators seem to overflow consistently. :)

0: jdbc:farrago:> alter system set "calcVirtualMachine"='CALCVM_JAVA';

No rows affected (0.123 seconds)

0: jdbc:farrago:> values 8876543210987654321 + 1000000000000000000;

+-----------------------+

|EXPR$0 |

+-----------------------+

| -8570200862721897295 |

+-----------------------+

1 row selected (0.118 seconds)

0: jdbc:farrago:> alter system set "calcVirtualMachine"='CALCVM_FENNEL';

No rows affected (0.055 seconds)

0: jdbc:farrago:> values 8876543210987654321 + 1000000000000000000;

+-----------------------+

|EXPR$0 |

+-----------------------+

| -8570200862721897295 |

+-----------------------+

1 row selected (0.952 seconds)