dynamobi/dynamobi — Issue #764: [FRG-106] expression with division and multiplication for Decimal(10,0) differs for java and fennel calc
Author: dynamobi-build
Created: 2011-12-30 03:47:49+00:00
State: open
Labels: FRG, Java Calc Codegen

[reporter="elin", created="Tue, 11 Apr 2006 14:35:04 -0500 (GMT-05:00)"]
An expression with decimal(10,0) will result in truncation if the java calc is used and rounding if fennel calc is used.

Division with 2 integers will result in truncation by both calcs.

Straight division of a decimal(10,0) by an integer doesn't round or truncate.

To reproduce on luciddb:

create table tdec (col1 decimal(10,0), col2 decimal(10,0), col3 decimal(10,0));

insert into tdec values

(101, 2, 7), (127, 2, 7);

alter system set "calcVirtualMachine"='CALCVM_FENNEL';

select col1 * col2 / col3 from tdec;

values (101 * 2 / 7);

select col1 / col3 from tdec;

alter system set "calcVirtualMachine"='CALCVM_JAVA';

select col1 * col2 / col3 from tdec;

values (101 * 2 / 7);

select col1 / col3 from tdec;