uobdic/bcfg2-setup — pull request #10 by mikewallace1979 (2013-01-03 15:04:28+00:00)
Title: Raise hadoop limits

Changes as requested by Royston:

1. You and I raised the file descriptor limit during a phone call before Christmas. We set the soft limit to 8192 and the hard limit to 16384. We’re still getting task trackers dying after about 36 hours of mapreduce job running. There aren’t many clues but what there is hints at the fd problem so could these be raised to 16384 and 32768

I think this is in /etc/security/limits.conf

2. There is a web page that suggests Hadoop needs per-process limits raised since the 2.6.27 kernel (DICE is on 32 I think): http://wiki.apache.org/hadoop/TooManyOpenFiles

It looks like max_user_instances is the default 128 on DICE still. At least there’s no file at /proc/sys/fs/epoll/max_user_instances

Can you add

fs.epoll.max_user_instances = 2048

to /etc/sysctl.conf

3. We were getting some crashing DataNodes caused I think by Out Of Memory errors which I fixed with the following three changes:

/etc/hadoop/hadoop-env.sh:

HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,DRFAS $HADOOP_DATANODE_OPTS –Xmx4g"

export HADOOP_CLIENT_OPTS="-Xmx1024m $HADOOP_CLIENT_OPTS"

/etc/hadoop/mapred-site.xml added:

mapred.child.java.opts

-Xmx1024m