hpcc-systems/HPCC-Platform — Issue #704: PIPE options possibly incomplete
Author: rpastrana
Created: 2011-10-17 21:15:45+00:00
State: open

The PIPE command allows the caller to specify the expected data format as CSV, or XML and seems to default to FLAT format if the format is not specified.
However, if the user attempts to pass the FLAT option, it is seen as a error by the syntax checker.

Since the PIPE command seems to process data as FLAT by default, it might not be incorrect to prevent the user from explicitly setting the type as FLAT, but it prevents the following code:

myPipeMacro(p, params,fileformat,layout) := MACRO
p := PIPE('callhdfs.sh ' + params + ' -format ' + #TEXT(fileformat), layout, fileformat);
ENDMACRO;

myPipeMacro(fred, '-x y', FLAT, MyLayout);

Conditionally setting passing the format param as follows doesn't pass syntax check either:
myPipeMacro(p, params,fileformat,layout) := MACRO
p := IF(#TEXT(HadoopFileFormat) != 'FLAT' OR #TEXT(HadoopFileFormat) != 'THOR',
PIPE('callhdfs.sh ' + params + ' -format ' + #TEXT(fileformat), layout),
PIPE('callhdfs.sh ' + params + ' -format ' + #TEXT(fileformat), layout, fileformat));
ENDMACRO;

Error: syntax error near "''" : expected CSV, GROUP, GROUPED, OUTPUT, REPEAT, XML (116, 30), 3002,
or
Error: Unknown identifier "FLAT" (128, 51), 2167,
Error: While expanding macro (128, 48), 2167,

@RussWhitehead