hpcc-systems/HPCC-Platform — pull request #2743 by ghalliday (2012-07-03 08:55:20+00:00)
Title: Introduce hash dedup helper functions 

Introduce extra hash-dedup helper functions to enable thor to optimize
the implementation. Also indicate whether the entire row is being
deduped - to avoid conversion to a key.

For hash dedup if the dedup criteria are much smaller than the full row
then it is worth saving just the dedup fields. However this may cause
extra work, may use more memory if the deduped rows are held in memory
(e.g., by a sort), or if there is little difference in the shrunk size.

To help this last criteria, this function allows thor to determine
whether or not removing some fields is likely to reduce the memory
usage.

See Issue #2700.

Signed-off-by: Gavin Halliday