Yelp/mrjob — Issue #257: --jobconf mapred.task.timeout does not appear to change timeout
Author: ekulakow
Created: 2011-11-08 18:34:17+00:00
State: open

Hi guys,

I've been running a decently large job that I'm having trouble completing because one of the reducers times out on the file upload part after it completes. I tried adding --jobconf mapred.task.timeout=1800000 to the command line, but it appears that the job terminated after 10 mins of uploading time anyways. I'm not sure why the upload should be having problems as it doesn't seem that the file is outrageously large, but it is significantly larger than the outputs of most of the other reducers (the key space is not well balanced - I'm going to try fixing this but it shouldn't matter for this bug). Anyways, it seems like mrjob is timing out stuff in the upload state, and that setting mapred.task.timeout to a larger number isn't listened to.

I'm on development-v0.2.x. Am using

bootstrap_actions: &boots
- s3://elasticmapreduce/bootstrap-actions/configurations/latest/memory-intensive

and

base_tmp_dir: /mnt/tmp/

As far as other flags go.

The bug was still happening without the memory-intensive bootstrap, though I didn't try changing mapred.task.timeout without it.

Some log file stuff:

2011-11-08 02:03:40,421 INFO org.apache.hadoop.mapred.Merger (main): Merging 9 sorted segments
2011-11-08 02:03:41,284 INFO org.apache.hadoop.mapred.Merger (main): Down to the last merge-pass, with 9 segments left of total size: 1648153512 bytes
2011-11-08 02:03:42,535 INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem (main): Creating new file 's3://path/to/output/part-00050' in S3
2011-11-08 02:03:42,537 INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem (main): Outputstream for key 'path/to/output/part-00050' writing to tempfile '/mnt/var/lib/hadoop/s3/output-5430952274196003800.tmp'
2011-11-08 02:03:42,578 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=1/0/0 in:0=1/92 [rec/s] out:0=0/92 [rec/s]
2011-11-08 02:03:42,579 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=10/0/0 in:0=10/92 [rec/s] out:0=0/92 [rec/s]
2011-11-08 02:03:42,579 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=100/0/0 in:1=100/92 [rec/s] out:0=0/92 [rec/s]

...

2011-11-08 02:11:14,944 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=9700000/210394/0 in:17798=9700000/545 [rec/s] out:386=210394/545 [rec/s]
2011-11-08 02:11:19,745 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=9800000/210394/0 in:17850=9800000/549 [rec/s] out:383=210394/549 [rec/s]
2011-11-08 02:11:24,922 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=9900000/210394/0 in:17870=9900000/554 [rec/s] out:379=210394/554 [rec/s]
2011-11-08 02:11:32,005 INFO org.apache.hadoop.streaming.PipeMapRed (main): R/W/S=10000000/210394/0 in:17793=10000000/562 [rec/s] out:374=210394/562 [rec/s]

End Of File

Anything else I can look at?