Implemented StatisticsUnpivot
* Input: ["schema", "stat1_col1", "stat2_col1", "stat3_col1", "stat1_col2", ...]
* Output: ["schema", "column", "converted", "stat1", "stat2", "stat3"]
Statistics functions as UDFs:
Currently using FieldReader to ensure consistent output type so that Unpivot doesn't get confused. All stats columns should be Nullable, so that stats functions can return NULL when N/A.
* custom versions of "count" that always return BigInt
* HyperLogLog based NDV that returns BigInt that works only on VarChars
* HyperLogLog with binary output that only works on VarChars
Add "append" concept to directory write.
* This is so stats can be stored in [table].stats.drill and be appended to be writing a new file into the directory.
AnalyzeTable functionality
* JavaCC syntax more-or-less copied from LucidDB.
* AnalyzePrule: DrillAnalyzeRel -> UnpivotPrel and StatsAggPrel
Add getDrillTable() to DrillScanRelBase and ScanPrel
* since ScanPrel does not inherit from DrillScanRelBase, this requires adding a DrillTable to the constructor
* This is done so that a custom ReflectiveRelMetadataProvider can access the DrillTable associated with Logical/Physical scans.
Attach DrillTableMetadata to DrillTable.
* DrillTableMetadata represents the data scanned from a corresponding ".stats.drill" table
* In order to avoid doing query execution right after the ".stats.drill" table is found, metadata is not actually collected until the MaterializationVisitor is used.
** Currently, the metadata source must be a string (so that a SQL query can be created). Doing this with a table is probably more complicated.
** Query is set up to extract only the most recent statistics results for each column.
Implement a RelMetadataProvider to hook DrillTableMetadata into Optiq
Also, modify WorkspaceSchemaFactory to add DrillTableMetadata to DrillTables it creates.