linkedin/datafu — pull request #61 by matthayes (2013-08-06 22:14:38+00:00)
Title: Coalesce should merge types where possible, add EmptyBagToNull

When using Coalesce on many fields it can be annoying to have to know whether, for example, the field is an int or a long. For example:

Coalesce(val1,0L) as val1,
Coalesce(val2,0L) as val2,
Coalesce(val3,0L) as val3,
Coalesce(val4,0L) as val4,

Suppose that among val1,val2,val3,val4, some are ints and some are longs. In this situation we may choose to make all types long for simplicity. Before this change this would fail, because int and long are different.

This adds a 'lazy' mode to Coalesce, where it tries to merge types if possible when there is incompatibility, and the value will be cast to this type. The default is 'strict' mode.

I also added EmptyBagToNull, which is especially useful at performing a "left join" using COGROUP:

result = FOREACH (COGROUP data_1 by id,data_2 BY id,data_3 BY id,data_4 BY id) generate
FLATTEN(data_1), -- left joining on this, flattening an empty bag removes the row
FLATTEN(EmptyBagToNull(data_2)),
FLATTEN(EmptyBagToNull(data_3)),
FLATTEN(EmptyBagToNull(data_4));