kijiproject/bento-cluster — pull request #6 by kimballa (2012-12-05 22:24:51+00:00)
Title: CLUSTER-9. bento/bento-env.sh do not work if invoked via symlinks.

Similar to SCHEMA-43, the bento scripts do not work correctly
if they are invoked via symlinks. They will not use the dirname of $0
correctly, so we need to traverse symlinks "manually".

The location of the bento-env.sh script (or symlink) is also tied to the
'build/' directory that the tasktracker puts its state under. This is
now moved under the 'state/' directory regardless of where you invoke
it from.

TEST PROTOCOL:

1) create symlinks to bin/bento and bin/bento-env.sh

aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ ln -s bento-cluster-1.0.0-rc2-SNAPSHOT/bin/bento bento
aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ ln -s bento-cluster-1.0.0-rc2-SNAPSHOT/bin/bento-env.sh bento-env.sh

aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ ls -l
total 12
lrwxrwxrwx 1 aaron aaron 42 Dec 5 14:13 bento -> bento-cluster-1.0.0-rc2-SNAPSHOT/bin/bento
drwxrwxr-x 5 aaron aaron 4096 Dec 5 14:13 bento-cluster-1.0.0-rc2-SNAPSHOT
lrwxrwxrwx 1 aaron aaron 49 Dec 5 14:13 bento-env.sh -> bento-cluster-1.0.0-rc2-SNAPSHOT/bin/bento-env.sh

2) start the bento cluster from "far away" using the symlinks. It looks like it's writing conf files to the right spot.

[aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bn-script)$ ./bento start
Configuring bento-cluster...
Checking if the Hadoop/HBase default ports are open...
All default ports open. Using these in the Hadoop/HBase configuration for your cluster.
Writing Hadoop configuration files to: /home/aaron/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release/bento-cluster-1.0.0-rc2-SNAPSHOT/bin/../lib/hadoop-2.0.0-mr1-cdh4.1.1/conf
Writing HBase configuration files to: /home/aaron/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release/bento-cluster-1.0.0-rc2-SNAPSHOT/bin/../lib/hbase-0.92.1-cdh4.1.1/conf
Configuration files successfully written!

After your clusters have (re)started, you can visit their web interfaces:
HDFS NameNode: http://localhost:50070
MapReduce JobTracker: http://localhost:50030
HBase Master: http://localhost:60010

Starting bento-cluster...
Waiting for clusters to start...
bento-cluster started.

3) verify that the "build/" directory, which holds legit state, is always trapped under $BENTO_CLUSTER_HOME/state/.

aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ ls
bento bento-cluster-1.0.0-rc2-SNAPSHOT bento-env.sh
aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ ls bento-cluster-1.0.0-rc2-SNAPSHOT/
bin cluster-conf conf lib LICENSE.txt NOTICE.txt README.md state
aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ ls bento-cluster-1.0.0-rc2-SNAPSHOT/state/
bento-cluster.pid build dfscluster userlogs zookeeper

4) use bento-env.sh via the symlink; observe that hadoop fs -ls, etc. all work.

aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ source ./bento-env.sh
Set BENTO_CLUSTER_HOME=/home/aaron/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release/bento-cluster-1.0.0-rc2-SNAPSHOT/bin/..
Set HADOOP_HOME=/home/aaron/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release/bento-cluster-1.0.0-rc2-SNAPSHOT/bin/../lib/hadoop-2.0.0-mr1-cdh4.1.1
Set HBASE_HOME=/home/aaron/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release/bento-cluster-1.0.0-rc2-SNAPSHOT/bin/../lib/hbase-0.92.1-cdh4.1.1
Added Hadoop, HBase, and bento-cluster binaries to PATH.
aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ hadoop fs -ls
Found 1 items
drwxr-xr-x - aaron supergroup 0 2012-12-05 14:16 hbase

aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ which hadoop
/home/aaron/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release/bento-cluster-1.0.0-rc2-SNAPSHOT/bin/../lib/hadoop-2.0.0-mr1-cdh4.1.1/bin/hadoop

5) Demonstrate that mapreduce still works even when forcing this build/ dir to go someplace nicer.

aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5 5000
Number of Maps = 5
Samples per Map = 5000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
12/12/05 14:18:12 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/12/05 14:18:12 INFO mapred.FileInputFormat: Total input paths to process : 5
12/12/05 14:18:12 INFO mapred.JobClient: Running job: job_20121205141656611_0001
12/12/05 14:18:13 INFO mapred.JobClient: map 0% reduce 0%
12/12/05 14:18:18 INFO mapred.JobClient: map 20% reduce 0%
12/12/05 14:18:19 INFO mapred.JobClient: map 40% reduce 0%
12/12/05 14:18:22 INFO mapred.JobClient: map 80% reduce 0%
12/12/05 14:18:24 INFO mapred.JobClient: map 100% reduce 0%
12/12/05 14:18:26 INFO mapred.JobClient: map 100% reduce 100%
12/12/05 14:18:27 INFO mapred.JobClient: Job complete: job_20121205141656611_0001
[snip]
Job Finished in 15.511 seconds
12/12/05 14:18:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Estimated value of Pi is 3.14208000000000000000

6) We're done here.

aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ ./bento stop
Stopping bento-cluster...
Waiting for shutdown...
bento-cluster shutdown.

aaron@topsail:~/src/kiji/bento-cluster/target/bento-cluster-1.0.0-rc2-SNAPSHOT-release (cluster-9-bin-script)$ jps
4080 org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar
15849 Jps