apache/spark — pull request #2082 by chuxi (2014-08-21 13:02:32+00:00)
Title: SPARK-2096 [SQL]: Correctly parse dot notations for accessing an array of structs

For example, "arrayOfStruct" is an array of structs and every element of this array has a field called "field1". "arrayOfStruct[0].field1" means to access the value of "field1" for the first element of "arrayOfStruct", but the SQL parser (in sql-core) treats "field1" as an alias. Also, "arrayOfStruct.field1" means to access all values of "field1" in this array of structs and the returns those values as an array. But, the SQL parser cannot resolve it.

I have passed the test case in JsonSuite ("Complex field and type inferring (Ignored)") which is ignored, by a little modified.
modified test part :
checkAnswer(
sql("select arrayOfStruct.field1, arrayOfStruct.field2 from jsonTable"),
(Seq(true, false, null), Seq("str1", null, null)) :: Nil
)
However, another question is repeated nested structure is a problem, like arrayOfStruct.field1.arrayOfStruct.field1 or arrayOfStruct[0].field1.arrayOfStruct[0].field1
I plan to ignore this problem and try to add "select arrayOfStruct.field1, arrayOfStruct.field2 from jsonTable where arrayOfStruct.field1==true "
Besides, my friend anyweil (Wei Li) solved the problem of arrayOfStruct.field1 and its Filter part( means where parsing).
I am fresh here but will continue working on spark :)

I checked the problem " where arrayOfStruct.field1==true "
this problem will lead to modify every kind of comparisonExpression. And I think it makes no sense to add this function. So I discard it.
Over.