hpcc-systems/HPCC-Platform — Issue #1045: Resolving Context Dependent expressions
Author: ghalliday
Created: 2011-11-22 10:46:04+00:00
State: open

The problem is that an expression may have a different meaning depending on where it is used, for example LEFT/RIGHT have a context sensitive meaning (e.g., two PROJECTS on the same record format).
At the moment LEFT is represented as no_left(record) - it has no unique sequence number.
So if you have a project on one record, with a project of the same record as a child operation within it the LEFT becomes ambiguous. This can cause particular problems if one is nested within another - because it means that an optimization similar to loop invariance could move an expression to a location where it is still valid, but with a different meaning.

My previous attempt at a solution was as follows:
Create a unique ID for each instance of LEFT, if it is inside a function add all the parameters as arguments.
When the tree is normalized replace the parameterised arguments with unique ids.

This almost works (but slows the code generator down). The disadvantage of that approach is that calling the same function with different arguments can create different instances, when they should be the same.
However...if I changed it so they were dependent on all parameters that were used by the input dataset, that
is almost certainly what the user expects.

There is an additional problem which is solved in SQL using aliases:
ds(ds.value > ave(ds, ds.value));
especially generated from expanding the code:
x(a) := ds(value > a);
x(ave(ds, value));

The problem is that the value ds.value has a different meaning because there are two independent uses of ds.

I should introduce the concept of an alias:
x(ave(alias(ds), value))
which would internal expand to
ds(ds.value > ave(alias(ds), alias(ds).value));

The code generator should also implicitly create an alias if a dataset used inside a nested context when normalizing the tree.

The disadvantage is that some cses might be lost by the creation of the alias.
e..g,
x(a) := ds(value > a);
x(ave(ds, value));
output(ave(ds, value));
is likely to calculate ave(ds, value) twice.

(richardkchapman: Losing cse in weird cases sounds fine to me (better than giving wrong answers in weird cases))

In conclusion I think the following might ensure all expressions have a unique meaning.
i) Make sure all LEFT are unique as outlined above (a variant of previous scheme).
ii) Allow ALIAS(ds) which works by creating a unique instance of ds.
iii) Automatically create an ALIAS when I see a dataset being used in a nested context.
iv) Tag XMLTEXT/MATCHTEXT with a unique id associated with the PARSE.
v) Extend code to gather dependent tables to also include the uids for PARSE etc.