apache/spark — pull request #2230 by cloud-fan (2014-09-01 12:33:25+00:00)
Title: SPARK-2096 Correctly parse dot notations

First let me write down the current `projections` grammar of spark sql:

expression : orExpression
orExpression : andExpression {"or" andExpression}
andExpression : comparisonExpression {"and" comparisonExpression}
comparisonExpression : termExpression | termExpression "=" termExpression | termExpression ">" termExpression | ...
termExpression : productExpression {"+"|"-" productExpression}
productExpression : baseExpression {"*"|"/"|"%" baseExpression}
baseExpression : expression "[" expression "]" | "(" expression ")" | ident | ...
ident : identChar {identChar | digit} | delimiters | ...
identChar : letter | "_" | "."
delimiters : "," | ";" | "(" | ")" | "[" | "]" | ...
projection : expression [["AS"] ident]
projections : projection { "," projection}

For something like `a.b.c[1]`, it will be parsed as:

But for something like `a[1].b`, the current grammar can't parse it correctly.
A simple solution is written in `ParquetQuerySuite#NestedSqlParser`, changed grammars are:

identChar : letter | "_"
baseExpression : expression "[" expression "]" | expression "." ident | "(" expression ")" | ident | ...
This works well, but can't cover some corner case like `select t.a.b from table as t`:

`t.a.b` parsed as `GetField(GetField(UnResolved("t"), "a"), "b")` instead of `GetField(UnResolved("t.a"), "b")` using this new grammar.
However, we can't resolve `t` as it's not a filed, but the whole table.(if we could do this, then `select t from table as t` is legal, which is unexpected)
My solution is:

dotExpressionHeader : ident "." ident
baseExpression : expression "[" expression "]" | dotExpressionHeader | expression "." ident | "(" expression ")" | ident | ...
I passed all test cases under sql locally and add a more complex case.
I'm not familiar with the latter optimize phase, please correct me if I missed something.