lalithsuresh/Scaling-HDFS-NameNode — pull request #6 by lalithsuresh (2011-12-11 10:46:20+00:00)
Title: MySQL access optimisations to helper classes and DBConnector

The following commit implements the following:

- Remove transactions and session flushes that surrounded all read requests to the DB.
- Move from a single session factory to a session factory pool, and have DBConnector.obtainSession() return a session from a random session factory.
- Each session factory now has its own connection object in the ClusterJ connection pool (using the clusterj.connection.pool.size property). This value is limited by the number of empty msqld/api slots in the MySQL cluster.

The above changes have shown significant improvements as measured by the LoadGenerator benchmark. Note that we are still limited by contention at the read/write locks in the rest of the Namenode code base.

$: hadoop org.apache.hadoop.fs.loadGenerator.LoadGenerator -numOfThreads 1 -elapsedTime 10 -startTime 1 -root /$NN -readProbability 0.80 -writeProbability 0.10

Average open execution time: 46.75333333333333ms
Average list execution time: 10.6ms
Average deletion execution time: 25.642857142857142ms
Average create execution time: 37.857142857142854ms
Average write_close execution time: 124.71428571428571ms

$: hadoop org.apache.hadoop.fs.loadGenerator.LoadGenerator -numOfThreads 5 -elapsedTime 10 -startTime 1 -root /$NN -readProbability 0.80 -writeProbability 0.10
Average open execution time: 177.1387283236994ms
Average list execution time: 88.64ms
Average deletion execution time: 169.43478260869566ms
Average create execution time: 104.43478260869566ms
Average write_close execution time: 479.1304347826087ms

Kindly review the changes and I'll merge with stable once you all ACK.