hpcc-systems/HPCC-Platform — pull request #2149 by rengolin (2012-04-23 11:59:04+00:00)
Title: Roxie dynamic files - DO NOT PULL

This pull request will completely break Roxie, but I need some serious code review.

The main points are:

1. Adding a dynamic flag resolution to QueryFactory. Done this way (instead of in RoxiePackage, as originally intented), because all access to the flag was done via queryF->RoxieP, ie. to avoid more indirections. I don't mind either way, if anyone think it's better to be in RoxiePackage, I can easily change that.
2. Common ```variableFileName``` in all activities to be set first by the dynamicResolution flag in the factories (on Base-most concrete class), and then by specific activity helpers' flags (such as ```TDXdynamicfilename```). Factories' flag is set by QueryFactory's flag on Base-most concrete class's constructor as well.
3. Setting the flag allows Roxie to resolve files dynamically as it would when ```#option ('allowVariableRoxieFilenames', 1);``` is set.

The main problems:

1. ```attachWorkUnit``` has been changed to also create if there was none. This should be split into two different methods, maybe.
2. ```getWorkUnitDll``` should know the difference between shared objects and executables, either by receiving this information or by inspecting the workunit (a new flag apart from @dll?)
3. Due to ```CQueryFactory``` constructor being called from numerous places, I've hacked in a dll check to make it dynamic, but this is wrong. I need to find out which places should have the flag set via the public method ```setDynamicFileResolution()```.

The main good things:

1. Now, ```runOnce``` can use both SocketListener and WorkunitListener, so we can limit the dynamic flag to workunits only (though this has not been done yet).
2. The dynamic resolution logic in activities and factories is consistent and very central, and the changes are minor in that area, so should not affect current behaviour much.

What I want to know?

1. This is a big change (for a first-timer in Roxie), so I need to make sure I'm on the right path before I consolidate more changes in that direction.
2. Not just general guidelines, but comments on architectural style would be welcomed.

This is the example I'm trying to run (the OVERWRITE makes it fail in Roxie):

filename := 'regress::roxie::fileaccess';
rec := record
string line;
end;
raw := dataset([{'foo'},{'bar'},{'baz'}], rec);
output(raw,,filename,CSV,OVERWRITE);
ds := dataset(filename, rec, CSV);
output(ds);

$ eclcc static.ecl
$ ./static DALISERVERS=localhost

This works with current patches.