hpcc-systems/HPCC-Platform — Issue #1169: ,EXPECTED & : EXPECTED
Author: dabayliss
Created: 2011-12-27 15:42:33+00:00
State: open

Thus far we have used ,FEW and MANY to indicate to the optimizer when in-memory solutions are possible and would give us significant lift.
However, if one is producing a LIBRARY of code that might be used across a wide range of machines (with big-mems ranging from a gig to tens of gigs - such as an ML library as an entirely random example) - the subjective and arbitrary nature of the names become a problem.
I can think of two solutions; one icky but effective - one elegant but possibly harder to do.
1) MyLibrary.BigMemSize - this would allow us to implement IF ( x < Y ) code to use the correct ,FEW or ,MANY option
2) Allow us to specify the EXPECTED number of records out of an attribute (either using TABLE( , expected(expressions)) or : EXPECTED(expected) - the get the optimizer to do the math to figure out the correct form of the component to use (as an added nicety - expected/actual results could be tracked in the wu)

David