dlyubimov/HBase-Lattice — Issue #6: Compiler's UDFs pull model server-side from hbase instead of client-side
Author: dlyubimov
Created: 2011-12-22 19:21:40+00:00
State: open

Currently, UDFs are configured with model string in the compiler body which is sent server-side. Compiler generator pulls model and then configures all UDFs with it.

This is obviously pretty expensive, because there may be too many 'define's in the compiler script each of which is passing the model to the function instance in the backend.

Instead, too much more scalable alternatives are available:
* UDF instances pull the model by name server side themselves;
* only one shared configuration property is passed (broadcasted) to all backend instances of all UDFs.

while the former option is attractive because of minimum of parameterization is required, it is not quite safe. If model update happens during compiler job setup, different backends (or even separate functions) may pull different version of a model, and then the whole result of compilation run may be inconsistent.

The latter option is probably much better because it ensures all UDF jobs within compiler run will have consistent view of the model.