amplab/shark — pull request #111 by rxin (2013-07-30 07:14:16+00:00)
Title: Shark Scala API update

@mateiz, @jhartlaub, @markhamstra

Please take a look at the new UI change. There are multiple changes:

1. TableRDD has a new field tableDesc, which includes some schema definition. Previously it included scheme description that uses some Hive internal data structure. This is now gone and replaced with Shark's own data structure to represent the types.

2. Added a executeSql command to SharkContext, which returns a tuple (Schema, ResultData).
Question 1: Should I return a ResultSet class with a scheme field and a data field rather than using Scala's tuple 2?
Question 2: Should executeSql replace the existing sql() call?

3. Added a bunch of methods in Row to deal with complex data types. The problem with those is they are exposing some internal Hive data structures, and those can change and are not very well documented. Should I just disable those for now until we can properly generate all data structures in Shark using code gen?

In the long run, I would like to make the Row class serializable and have a proper equals / hashcode implementation, so we can run all Spark transformations/actions on those (collect, reduce, etc). But we won't be able to do that until we do full code generation for all operator data structures in Shark.