dynamic hadoop tuning in MyHadoop

by checking the job-environment, a lot of tuning can be done to Hadoop to make it run better on the fatter nodes available in HPC systems; here some steps are implemented based upon SLURM variables, but they all have been done in the conf file and can easily be implemented with other resource managers...
ack, fixed now in new commit;

thanks!
Hugo


On Sun, Jun 8, 2014 at 10:41 PM, Glenn K. Lockwood <notifications@github.com
> wrote:

> In bin/myhadoop-configure.sh:
>
> > @@ -226,6 +227,12 @@ config_subs[MAPRED_LOCAL_DIR]="$MH_SCRATCH_DIR/mapred_scratch"
> >  config_subs[HADOOP_TMP_DIR]="$MH_SCRATCH_DIR/tmp"
> >  config_subs[DFS_NAME_DIR]="$MH_SCRATCH_DIR/namenode_data"
> >  config_subs[DFS_DATA_DIR]="$MH_SCRATCH_DIR/hdfs_data"
> > +config_subs[DFS_REPLICATION]="$MH_DFS_REPLICATION"
> > +config_subs[DFS_BLOCK_SIZE]="$MH_DFS_BLOCK_SIZE"
> > +config_subs[MAPRED_TASKTRACKER_MAP_TASKS_MAXIMUM]="$MH_MAP_TASKS_MAXIMUM"
> > +config_subs[MAPRED_TASK_TRACKER_REDUCE_TASKS_MAXIMUM]="$MH_REDUCE_TASKS_MAXIMUM"
>
> This should be MAPRED_TASKTRACKER_REDUCE_TASKS_MAXIMUM; the current
> version leaves the following in mapred-site.xml after myhadoop-configure.sh
> is run:
>
> <property>
>   <name>mapred.tasktracker.reduce.tasks.maximum</name>
>   <value>MAPRED_TASKTRACKER_REDUCE_TASKS_MAXIMUM</value>
>   <description>The MH_REDUCE_TASKS_MAXIMUM will set the maximum amount
>      of REDUCE tasks to be started on a single TASK node, this is most
>      likely memory bound.</description>
> </property>
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/glennklockwood/myhadoop/pull/8/files#r13526090>.
>