shravanmn/Yahoo_LDA — Issue #6: unable to run formatter
Author: josephykwang
Created: 2012-01-06 00:11:49+00:00
State: open

I run

./runLDA.sh 1 "data" train default "/user/jwang/simulation/data/offers/current" "/user/jwang/workspace/lda/" 3244 1000 5 /user/jwang/software/YLDA/LDALibs.jar 20 "/user/jwang/workspace/ldatemp"

There are text files in /user/jwang/simulation/data/offers/current, where each line is a document.

It invoked a streaming job, where mapper run successfully. However, the reducers failed. Here is the error associated with one of reducers:

java.io.IOException: subprocess exited with error code 1
R/W/S=540/0/0 in:77=540/7 [rec/s] out:0=0/7 [rec/s]
minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
HOST=null
USER=root
HADOOP_USER=null
last Hadoop input: |null|
last tool output: |null|
Date: Thu Jan 05 16:04:36 PST 2012
Broken pipe
at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:131)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:469)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)

Any idea?