remove extra dependency

RB_ID=98886
Make sure new log id not conflict with existing file
   In createNewLog(), only one directory is searched to check for duplicate log id. Then the id is used to overwrite lastLogId.
Given that lastLogId may not be updated if the directory was almost full, which later may be freed up, this may cause the id duplicate with some log file.
It looks like regression from BOOKKEEPER-345.

RB_ID=98074
Hedwig multi-threaded readahead cache support:

This change introduces multiple-threads support for hedwig-server readahead cache,  each topic uses affinity of one thread in the threads pool. Further, guava caching is used to optimize for server work load, instead of the single-thread map in the past. (In single thread test cases, the performance improvement is not observable).

By default, (# of processors) threads are allocated to serve the readahead cache.

Note that it is possible (though unlikely) that stub entry can get evicted by LRU policy, the callback will be notified with a distinct exception (newly introduced).

The per-ledger entry tracking map was to allow entries consumed to be removed aggressively.
Given that guava cache already have the lru policy, which the new design leverage, it is less an issue.
Expire-after-access will adds overhead to the cache, and it does not seem worth to use for this purpose.

RB_ID=98825
Use soft reference and clean up (be paranoid to enqueue stub-value eviction to thread pool, to reduce cache contention)

RB_ID=100454
Remove cache value weight hack

RB_ID=101228
FileInfo race condition:

Address ledger index fileInfo race condition between concurrent reader and move-to-new-location, by adding file-channel synchronization.
Given that the index file-channel is only used (by reader) to load pages which are not in cache, this synchronization should not impact performance significantly. (Long term, it is better to get rid of index file)

Added check of same path of new location. Error should be thrown, unfortunately some tests depend on this not checked here.

RB_ID=101452
fix TestTopicBasedLoadShedder test failures
- fix TestTopicBasedLoadShedder test failures

1) numTopics in HubLoad has been changed to AtomicInteger. but the test case still assume it was primitive integer. it fails most of test cases in this file.
2) different number of topics are used for initialization and LoadShedder. so the test cases doesn't work correctly.
3) replace bitwise & with logical &&

RB_ID=101242
merge opensource changes
- BOOKKEEPER-397: Make the hedwig client in RegionManager configurable. (Aniruddha via sijie)
- BOOKKEEPER-388: Document bookie format command (kiran_bc via ivank) [Corrected CHANGES.txt]
- BOOKKEEPER-388: Document bookie format command (vinayrpet via ivank)
- BOOKKEEPER-405: Let's add Thread name for ReplicationWorker thread. (umamahesh via ivank)
- BOOKKEEPER-252: Hedwig: provide a subscription mode to kill other subscription channel when hedwig client is used as a proxy-style server. (sijie via ivank)
- BOOKKEEPER-32: Clean up LOG.debug statements (Stu Hood via sijie)
- BOOKKEEPER-403: ReReadMetadataCb is not executed in the thread responsible for that ledger (ivank)
- BOOKKEEPER-392: Racey ConcurrentMap usage in java hedwig-client (Stu Hood via sijie)
- BOOKKEEPER-325: Delay the replication of a ledger if RW found that its last fragment is in underReplication. (umamahesh via ivank)

RB_ID=101267
merge open source changes to latest (includes part2 (multiplexing) & part3)

- BOOKKEEPER-475: BookieRecoveryTest#testSyncBookieRecoveryToRandomBookiesCheckForDupes() iterates too much (ivank via fpj)
- BOOKKEEPER-461: Delivery throughput degrades when there are lots of publishers w/ high traffic. (sijie via ivank)
- BOOKKEEPER-470: Possible infinite loop in simple.SubscribeReconnectCallback (sijie via ivank)
- BOOKKEEPER-399: Let hub server configure write quorum from ack quorum. (sijie via fpj)
- BOOKKEEPER-484: Misc fixes for test scripts (ivank via fpj)
- BOOKKEEPER-347: Provide mechanism to detect r-o bookie by the bookie clients (Vinay via ivank)
- BOOKKEEPER-442: Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges. (jiannan via ivank)
- BOOKKEEPER-459: Rename metastore mock implementation to InMemory implementation (jiannan via ivank)
- [REVERT] BOOKKEEPER-336 bookie readEntries is taking more time if the ensemble has failed bookie(s) Basic speculative functionality  in place Accidently committed this change, before approval. Reverting (ivank)
- BOOKKEEPER-291: BKMBeanRegistry uses log4j directly (fpj via ivank)
- fix test failures after merging
- BOOKKEEPER-336 bookie readEntries is taking more time if the ensemble has failed bookie(s) Basic speculative functionality  in place
- BOOKKEEPER-351: asyncAddEntry should not throw an exception (Matteo Merli via sijie)
- BOOKKEEPER-485: TestFencing hung (ivank via fpj)
- BOOKKEEPER-482: Precommit is reporting findbugs errors in trunk (ivank via sijie)
- BOOKKEEPER-389: add documentation for message filter. (sijie via ivank)
- BOOKKEEPER-480: Fix javac warnings (ivank via sijie)
- BOOKKEEPER-481: Fix javadoc warnings (ivank via sijie)
- BOOKKEEPER-487: Add existed hub server settings to configuration template file (sijie via ivank)
- BOOKKEEPER-457: Create a format command for Hedwig to cleanup its metadata. (sijie via ivank)
- BOOKKEEPER-476: Log to file during tests (ivank via fpj)
- BOOKKEEPER-477: In ReadOnlyBookieTest, we should wait for the bookie to die before asserting on it (ivank via fpj)
- BOOKKEEPER-479: Fix apache-rat issues in tree (ivank via fpj)
- BOOKKEEPER-471: Add scripts for preCommit testing (ivank)
- BOOKKEEPER-440: Make Write/Delete SubscriptionData Restricted to Version (Fangmin Lv via ivank)
- BOOKKEEPER-460: LedgerDeleteTest checks wrong place for log file (Fangmin Lv via ivank) [missing file]
- BOOKKEEPER-460: LedgerDeleteTest checks wrong place for log file (Fangmin Lv via ivank)
- BOOKKEEPER-467: Allocate ports for testing dynamically (ivank)
- BOOKKEEPER-454: hedwig c++ tester script assumes sh is bash (ivank)
- BOOKKEEPER-466: ZooKeeper test utility sets the port number as the tickTime (ivank)
- BOOKKEEPER-468: Remove <echo> from protobuf generation in hedwig (ivank)
- BOOKKEEPER-204: Provide a MetaStore interface, and a mock implementation. (Jiannan Wang via ivank)
- BOOKKEEPER-444: Refactor pending read op to make speculative reads possible (ivank)
- BOOKKEEPER-452: Rename ClientConfiguration multiplexing_enabled to subscription_connection_sharing_enabled (sijie via ivank)
- BOOKKEEPER-370: implement multiplexing cpp client. (sijie via ivank)
- BOOKKEEPER-368 Implementing multiplexing java client. (sijie via ivank)
- BOOKKEEPER-434: [Hedwig CPP Client] Delay resolving default host until necessary. (sijie via ivank)
- BOOKKEEPER-425: Cleanup Bookie id generation (ivank via fpj)
- BOOKKEEPER-439: No more messages delivered after deleted consumed ledgers. (sijie via ivank)
- BOOKKEEPER-346: Detect IOExceptions in LedgerCache and bookie should look at next ledger dir(if any) (Vinay via ivank)
- BOOKKEEPER-427: TestConcurrentTopicAcquisition hangs every so often (ivank)
- BOOKKEEPER-411: Add CloseSubscription Request for multiplexing support (sijie via ivank)
- BOOKKEEPER-435: Create SubscriptionChannelManager to manage all subscription channel (sijie via ivank)
- BOOKKEEPER-345: Detect IOExceptions on entrylogger and bookie should consider next ledger dir(if any) (Vinay via ivank)
- BOOKKEEPER-369: re-factor hedwig cpp client to provide better interface to support both one-subscription-per-channel and multiple-subscriptions-per-channel. (sijie via ivank)
- BOOKKEEPER-422: Simplify AbstractSubscriptionManager (stu via fpj)
- BOOKKEEPER-413: Hedwig C++ client: Rename RUN_AS_SSL_MODE to SSL_ENABLED (ivank via sijie)
- BOOKKEEPER-415: Rename DeliveryThrottle to MessageWindowSize (ivank via sijie)
- BOOKKEEPER-315: Ledger entries should be replicated sequentially instead of parallel. (umamahesh via ivank)
- BOOKKEEPER-319: Manage auditing and replication processes (Vinay via ivank)
- BOOKKEEPER-367: Server-Side Message Delivery Flow Control (sijie via ivank)
- BOOKKEEPER-143: Add SSL support for hedwig cpp client (sijie via ivank)
- BOOKKEEPER-364: re-factor hedwig java client to support both one-subscription-per-channel and multiplex-subscriptions-per-channel. (sijie via ivank)

RB_ID=102058
Fix the read only bookie test
The no write-able ledger directories exception is treated specially upstream so we should not mask it and let it be handled by the caller so that read only flag may be set when read only mode has been enabled.

RB_ID=98818
Separate vector clock of message-specific and region specific, refer to BOOKKEEPER-499 design document for details.

RB_ID=103508
TestUpdateSubscriptionState hangs and TestBackwardCompat fails.
- TestUpdateSubscriptionState hangs due to an operation doesn't callback when release topic. That opration blocks another opration to execute. so the tests failed to wait the response and hang.
- Since the protocol changes introduced in BookKeeper breaks the compat between a new bookkeeper client and an old bookie server, it breaks the TestBackwardCompat test case. No good idea to fix it, just ignore now.

RB_ID=104506
fix findbugs warnings in bookkeeper-server module.
- add Serializable for Comparator.
- ignore bytes noise warnings for using a slab to store references for bytes.
- final for static variables.

RB_ID=115262
fix findbugs warning in hedwig-client & hedwig-server modules.

    - remove unread fields.
    - ensure same variable are all under synchronization, not just %64 synchronization.
    - improve ServerStats to ensure OpStatData expose the metrics thru JMX.

RB_ID=104758
fix ledger index file fd leaking, which is cased by we don't release ref count when we use a file info.

RB_ID=119035
Add approximate weight for stub entry, which will be corrected once the entry replaced with actual value. The idea is to check and balance # of stubs if there is memory pressure on the read-ahead cache.

RB_ID=119297
Rename SkipListLedgerStorage to SortedLedgerStorage, per external feedback.

RB_ID=119264
Shutdown wait for executor and flush drains mem-table.
This change cover following items:
1. Ledger store shutdown wait for executor up to 3 seconds before forcing termination.
2. Ledger store flush should drain mem-table (derived class missing flush override).

RB_ID=119466