nathanmarz/dfs-datastores — pull request #48 by abolibibelot (2014-05-22 13:58:56+00:00)
Title: checkValidStructure fails when using yarn

The temporary directory hierarchy has changed between mr v1 and YARN. We run the same jobs on both frameworks and map outputs are typically laid out the following way:

Hadoop MR v1:
hdfs://root/tmp/foo/_temporary/_attempt_201405190855_34040_m_000005_0/2014-05-19/13/EU/part-000050

Hadoop YARN:
hdfs://root/tmp/foo/_temporary/1/_temporary/attempt_1398254330120_19508_m_000000_0/2014-05-19/13/EU/part-000000

The code assumed pruning hadoop specific directories was a matter of dropping every directory starting with _, from the beginning onwards, and stopping at the first non _ prefixed dir. It's no longer true when using yarn (along with cascading in our test case). With YARN the check fails systematically.