twitter/scalding — Issue #3: Error when outputting a local job to a file that sits in the current directory
Author: echen
Created: 2012-01-13 19:01:06+00:00
State: open

I can run `scripts/scald.rb --local tutorial/Tutorial0.scala ` fine as is (from the root of the scalding/ repo). However, if I change
val output = TextLine("tutorial/data/output0.txt")
to
val output = TextLine("output0.txt")
I get this error:

compiling tutorial/Tutorial0.scala
12/01/13 10:49:38 INFO util.Version: Concurrent, Inc - Cascading 2.0.0 [hadoop-0.20.2+]
12/01/13 10:49:38 INFO flow.Flow: [] starting
12/01/13 10:49:38 INFO flow.Flow: [] source: FileTap["TextLine[['num', 'line']->[ALL]]"]["tutorial/data/hello.txt"]"]
12/01/13 10:49:38 INFO flow.Flow: [] sink: FileTap["TextLine[['num', 'line']->[ALL]]"]["output0.txt"]"]
12/01/13 10:49:38 INFO flow.Flow: [] parallel execution is enabled: true
12/01/13 10:49:38 INFO flow.Flow: [] starting jobs: 1
12/01/13 10:49:38 INFO flow.Flow: [] allocating threads: 1
12/01/13 10:49:38 INFO planner.FlowStep: [] starting step: (1/1) local
12/01/13 10:49:38 ERROR local.LocalStepRunner: unable to prepare operation graph
java.lang.NullPointerException
at cascading.tap.local.FileTap.openForWrite(FileTap.java:108)
at cascading.tap.local.FileTap.openForWrite(FileTap.java:48)
at cascading.flow.stream.SinkStage.prepare(SinkStage.java:57)
at cascading.flow.stream.StreamGraph.prepare(StreamGraph.java:147)
at cascading.flow.local.LocalStepRunner.call(LocalStepRunner.java:84)
at cascading.flow.local.LocalStepRunner.call(LocalStepRunner.java:41)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
12/01/13 10:49:39 WARN flow.Flow: stopping jobs
12/01/13 10:49:39 INFO planner.FlowStep: [] stopping: (1/1) local
12/01/13 10:49:39 WARN flow.Flow: stopped jobs
12/01/13 10:49:39 WARN flow.Flow: shutting down job executor
12/01/13 10:49:39 WARN flow.Flow: shutdown complete
Exception in thread "main" cascading.flow.FlowException: local step failed
at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:161)
at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:125)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:112)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:39)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.NullPointerException
at cascading.tap.local.FileTap.openForWrite(FileTap.java:108)
at cascading.tap.local.FileTap.openForWrite(FileTap.java:48)
at cascading.flow.stream.SinkStage.prepare(SinkStage.java:57)
at cascading.flow.stream.StreamGraph.prepare(StreamGraph.java:147)
at cascading.flow.local.LocalStepRunner.call(LocalStepRunner.java:84)
at cascading.flow.local.LocalStepRunner.call(LocalStepRunner.java:41)
... 5 more

I get this error in my own jobs as well (it's not just Tutorial0.scala).

There's no problem if I change the output to
val output = TextLine("foo/output0.txt")
(If the `foo` directory doesn't exist, the job simply creates it and dumps the output into `output0.txt` there.)
There also no problem if I use the absolute path to my current directory.

So it seems like the exception depends on whether the output filename contains "/" or not.