Cascading/lingual — pull request #18 by choss (2014-02-25 11:32:18+00:00)
Title: Fix for using lingual with tcsv directories on Hadoop cluster with part files in the directory

In regards to the discussion in the lingual user group here: https://groups.google.com/forum/#!topic/lingual-user/7-GzLgTO7PU

The summary of the topic is:
Suppose we have and example structure (created via cascading job) like below on the hadoop cluster:

/logfiles/D2014_02_18/service1.tcsv/part-00000-00001
/logfiles/D2014_02_18/service1.tcsv/part-00001-00000
/logfiles/D2014_02_18/service1.tcsv/part-00002-00021
/logfiles/D2014_02_18/service1.tcsv/part-00004-00005

/logfiles/D2014_02_18/service2.tcsv/part-00000-00001
/logfiles/D2014_02_18/service2.tcsv/part-00001-00000
/logfiles/D2014_02_18/service2.tcsv/part-00002-00021
/logfiles/D2014_02_18/service2.tcsv/part-00004-00005

We then invoke the lingual shell like this: pc:~/test$ lingual shell --config schemas=/logfiles/D2014_02_18 --platform hadoop --resultPath=/results --verbose

Lingual correctly enumerates all tables and creates the tables service1 and service2 in the D2014_02_18 schema. Unfortunately it does not read the TCSV information from the part file and therefore does not allow any operation on those files.

I have created a fix for this which only works (and is only supposed to work) on the hadoop platform.
The test case had to be changed a bit, since the result comparison relied on the order of the data in the resultset. Unfortunately I do not know how to restrict the test only to work with the hadoop platform, therefore this is not implemented. The test is fully green otherwise