Bug fix for caching of discovered HAGlue interfaces.

See https://sourceforge.net/apps/trac/bigdata/ticket/687 (HAJournalServer Cache not populated)

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7191 8f7bc0f5-282e-0410-95e3-8d296e9bb460
javadoc

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7193 8f7bc0f5-282e-0410-95e3-8d296e9bb460
edits pertaining to [1]. These edits are mainly javadoc, making fields private, and some code clarification.

[1] http://sourceforge.net/apps/trac/bigdata/ticket/687 (HAJournalServer Cache not populated)


git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7194 8f7bc0f5-282e-0410-95e3-8d296e9bb460
AbstractJournal, HAJournal, HAJournalServer: Added maximum clock skew configuration option.

Use of debug flag in HAJournal.getExtendedStatus() and HAStatusServletUtil.

Added test suite for clock skew.

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7195 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Missed class in the previous commit.

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7196 8f7bc0f5-282e-0410-95e3-8d296e9bb460
I have modified HALogNexus to NOT add an empty HALog file on startup to the in-memory index iff it is the last HALog file in commit counter sequence for that service.

I have written a unit test for this behavior (TestHA3JournalServerWithHALogs.test_startABC_emptyLogFileDeletedOnRestartC()).  I have verified that the test replicates the problem reported here and that the change to HALogNexus causes the test to run green.

I found and fixed a bug in the test harness where copyFiles() would not actually copy a single file (the fix was an update to the documentation).  I also exposed the copyFile() method to subclasses.

Note: This commit does not include the test suite.  I need to vet some other edits in the test suite before I can commit that code. However, this commit does fix the problem desribed for this ticket.

@see http://sourceforge.net/apps/trac/bigdata/ticket/679 (HAJournalServer can not restart due to logically empty log file)


git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7197 8f7bc0f5-282e-0410-95e3-8d296e9bb460
javadoc only.

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7198 8f7bc0f5-282e-0410-95e3-8d296e9bb460
This commit includes the test suite for this ticket:

AbstractHA3BackupTestCase: lifted out the logic to compute a snapshot time such that the snapshot can not run during the unit test.

AbstractHA3JournalServerTestCase: fixed documentation on copyFiles() (it does not copy an individual file) and exposed copyFile() (which does copy an individual file).

AbstractHAJournalServerTestCase: Added method to do "COUNT(*)" query (and task for running that query).

StressTestHA3JournalServer: Added a stress test test suite and put the DROP ALL + LOAD ACID stress test in there.  We need to refactor other stress tests into this class as well.

TestAll: added references to the new tests.

TestHA3JournalServerWithHALogs: new test for [1].

TestHA3SnapshotPolicy2: logic to compute the "never run" snapshot time was lifted out of this class.

[1] http://sourceforge.net/apps/trac/bigdata/ticket/679 (HAJournalServer can not restart due to logically empty log file)



git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7199 8f7bc0f5-282e-0410-95e3-8d296e9bb460
reduced stress level for the ACID DROP ALL + LOAD test.

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7201 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Bug fix for [1]

[1] https://sourceforge.net/apps/trac/bigdata/ticket/692 (Failed to re-interrupt thread)

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7202 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Updating the river/jini dependencies to 2.2.1.  This provides a fix for [1,2] and [3] (I have not tested the DGC leak fix since we removed most of our dependencies on this feature).  DGC thread leaks have shown up in a number of contexts, including Futures for data service and HAJournalServer RMI methods.

[1] http://sourceforge.net/apps/trac/bigdata/ticket/646
[2] https://issues.apache.org/jira/browse/RIVER-416

[3] http://sourceforge.net/apps/trac/bigdata/ticket/678 (DGC Thread Leak)

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7203 8f7bc0f5-282e-0410-95e3-8d296e9bb460
There are a few locations in RemoteRepositoryManager which have this issue.  I have written a test in which the namespace contains characters that are not allowed in a URL without encoding and can demonstrate failures against that test.  There are also places on the server where the namespace is not being encoded when generating a URL, e.g., for the VOID description of the available KBs and also it was failing to decode the namespace in BigdataRDFServlet#getNamespace().

Changes are to:

- BigdataRDFServlet
- VoID
- MultiTenancyServlet
- ConnectOptions
- RemoteRepositoryManager
- TestMultiTenancyServlet

Commit is against both the 1.2.x and the READ_CACHE branches.

@see https://sourceforge.net/apps/trac/bigdata/ticket/689 (Missing URL encoding in RemoteRepositoryManager)

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7204 8f7bc0f5-282e-0410-95e3-8d296e9bb460
A CI deadlock has been observed again for [1].  I am modifying the test to use a timeout (3 minutes).  This way the test should fail if a transaction deadlock arises rather than causing CI to deadlock.

[1] https://sourceforge.net/apps/trac/bigdata/ticket/237 (CI deadlock in com.bigdata.concurrent.TestLockManager.test_multipleResourceLocking_resources10_locktries10)


git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7205 8f7bc0f5-282e-0410-95e3-8d296e9bb460
See [1,2]. Reconciled edits with Martyn, primarily with respect to (a) elision of pipeline change events in QuorumPipelineImpl to resolve a deadlock [2]; and (b) closing a concurrency hole when a service joins with a met quorum after the GATHER and before the PREPARE.
{{{
- QuorumPipelineImpl : incorporated changes for event elision. This
  provides a fix for <a
  href="http://sourceforge.net/apps/trac/bigdata/ticket/681" >
  HAJournalServer deadlock: pipelineRemove() and getLeaderId() </a>.
  I also fixed whitespace formatting for the indentation in
  elideEvents().  This was some odd mixture of tabs and spaces that
  was causing indent rendering issues.

- BufferedWrite: Ignored add of unused logger.

- WriteCacheService : did not accept log @ ERROR messages that were
  designated as "REMOVE TRACE".

- HA messages: modified some IHAMessages that did not override
  toString() to provide useful representations in log output.

- IHAAwaitServiceJoinRequest : javadoc and copyright notice.

- IHAWriteMessage : javadoc update for the commit counter semantics.

- HAReceiveService: added readFuture reference into the log message
  for changeDownstream.

- HASendService: incorporated tryno information into channel reopen
  log messages per mgc.

- HAGlue : folded in changes to awaitServiceJoin(), specifically it
  will now return the most recent consensus release time on the
  leader.  This closes a hole when a service joins after the GATHER
  and before the PREPARE.  By using the leader's consensus release
  time, the service will not permit transactions to start against a
  commit point that has been recycled by the leader.

- AbstractJournal: reconciled.

  - commitNow() was reusing the nonJoinedServices definition from the
    GATHER.  I added a JoinedAndNonJoinedServices helper class.
    Distinct instances of this class are now used for the GATHER and
    for the PREPARE/COMMIT.

  - doLocalCommit(): I did not pick up the 2nd half of this
    if/then/else.  Why is it there?

            if ((shouldFlush || true) && doubleSync) {

                _bufferStrategy.force(false/* metadata */);

            } else {
            	if (log.isInfoEnabled())
            		log.info("ALWAYS FORCE TO DISK");
                _bufferStrategy.force(false/* metadata */);
            }

- AbstractHATransactionService: changed updateReleaseTime() method
  into an override of setReleaseTime() which is public (versus
  protected) in order to expose this to the HAJournalServer.

- Journal: 

    - accepted implementation of updateReleaseTime() method from
      AbstractHATransactionService.

    - modified runWithBarrierLock() to log around the critical section
      so we can observe when this section runs and finishes.

- HAJournal: Modified to log @ INFO the request and result for an HA
  digest.  

- HAJournalServer: reconciled all changes.

  - ServiceLeaveTask: Yes, it should use quorum.token(), not
    NO_QUORUM. There is a difference between the quorum token (which
    just reflects zookeeper) and the journal token (which reflects
    zookeeper plus whether or not the local journal is HAReady).
  
                journal.setQuorumToken(getQuorum().token()); // This is correct.

		   - vs -

                journal.setQuorumToken(Quorum.NO_QUORUM); // This is wrong.

- TestHA3JournalServer: reconciled numerous changes.

  - Modified code that was using a sleep to (presumably) wait until
    the current quorum broke to instead just await the next quorum
    meet.

//        Thread.sleep(50); // wait a while for A to shutdown
//
//      final long token2 = quorum.awaitQuorum(awaitQuorumTimeout * 2,
//      TimeUnit.MILLISECONDS);

        // Wait for the next quorum meet on (token+1).
        final long token2 = awaitNextQuorumMeet(token);
    
- AbstractHA3JournalServerTestCase: converted stderr to log @ INFO.

- AbstractHAJournalServerTestCase: accepted log message @ INFO, but
  made it conditional.

Test failures:

- The 4 "overrides" tests need to be revisited. They still fail.

- testQuorumABC_HAStatusUpdatesWithFailovers: The problem appears to
  be that [leader] can not be compared with [serverB] and [serverC]
  using reference testing (==). [leader] has the same data in this
  case as [serverC] (same UUID, same TcpEndpoint).

  junit.framework.AssertionFailedError: Did not elect leader consistent with expectations:
   leader=Proxy[HAGlue,BasicInvocationHandler[BasicObjectEndpoint[b44e72e1-dd5c-4dbc-9640-129bdab11007,TcpEndpoint[192.168.1.135:55983]]]],
  serverB=Proxy[HAGlue,BasicInvocationHandler[BasicObjectEndpoint[073e0614-26a6-49be-83f4-381ce6338306,TcpEndpoint[192.168.1.135:55965]]]],
  serverC=Proxy[HAGlue,BasicInvocationHandler[BasicObjectEndpoint[b44e72e1-dd5c-4dbc-9640-129bdab11007,TcpEndpoint[192.168.1.135:55983]]]]
	at junit.framework.Assert.fail(Assert.java:47)
	at com.bigdata.journal.jini.ha.TestHA3JournalServer.testQuorumABC_HAStatusUpdatesWithFailovers(TestHA3JournalServer.java:2946)

- testStressStartAB_C_MultiTransactionResync_200_5: I have observed a
  failure of this test (after 33 out of 50 trials). A subsequent run
  of 50 trials succeeded.  Good, but not perfect.

  junit.framework.AssertionFailedError: Fail after 33 trials : java.util.concurrent.TimeoutException
	at junit.framework.TestCase2.fail(TestCase2.java:90)
	at com.bigdata.journal.jini.ha.TestHA3JournalServer.testStressStartAB_C_MultiTransactionResync_200_5(TestHA3JournalServer.java:620)

- testStress_RebuildWithPipelineReorganisation: failed on 7th run.

  java.lang.RuntimeException: junit.framework.AssertionFailedError
	at com.bigdata.io.TestCase3.assertCondition(TestCase3.java:250)
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase.awaitFullyMetQuorum(AbstractHA3JournalServerTestCase.java:1990)
	at com.bigdata.journal.jini.ha.TestHA3JournalServer.testStartABC_RebuildWithPipelineReorganisation(TestHA3JournalServer.java:1071)
	at com.bigdata.journal.jini.ha.TestHA3JournalServer._testStress_RebuildWithPipelineReorganisation(TestHA3JournalServer.java:1090)
}}}

[1] https://sourceforge.net/apps/trac/bigdata/ticket/530 (Journal HA)
[2] https://sourceforge.net/apps/trac/bigdata/ticket/681  (HAJournalServer deadlock: pipelineRemove() and getLeaderId())

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7206 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Updated assert in WriteCache per Martyn and documented the basis for the new assert.

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7207 8f7bc0f5-282e-0410-95e3-8d296e9bb460
syntax fix in exception message.

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7208 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Changes to the GATHER/PREPARE protocol in order to close some gaps
introduced by the recent refactoring to handle a service that joins
after the GATHER but before the PREPARE.
{{{
- Modified the GatherTask to return the consensusReleaseTime. This
  makes it possible to further validate things in Prepare2Phase.

- Modified the PREPARE message to include (a) whether the service
  participated in the GATHER; and (b) the consensusReleaseTime from
  the leader. This change is NOT backwards compatible. Migration
  requires that commits are suspended on the HA replication cluster
  during a rolling upgrade.  The decision to not provide backwards
  compatibility was made on the basis that we do not yet have a GA
  release with support for the HA replication cluster.

- Modified commitNow() to collect and pass along the set of joined
  services in the GATHER as well as in the PREPARE.

- Modified QuorumCommitImpl to pass along whether or not a service was
  joined in the GATHER as well as in the PREPARE and the consensus
  release time for the GATHER.

- Modified Prepare2Phase to check additional conditions based on
  whether the service participated in the GATHER and to verify the
  release time on the followers (must be consisent with the GATHER
  even if the follower joined after the GATHER).

- Modified HAJournal.config to use the FEDNAME environment variable.

- Updated comments in HAJournal.env.

- Isolated the FEDNAME as a static constant in the HA CI test suite.  

  ---- test results ----

- 2 unexpected failures. both are stochastic.

- testQuorumBreaksABC_failLeader: Failed once. Failed twice (uncurable
  service joined problem in seek consensus on C). 3rd time passed. 4th
  time passed.

java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at com.bigdata.io.TestCase3.assertCondition(TestCase3.java:233)
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase.awaitNextQuorumMeet(AbstractHA3JournalServerTestCase.java:2019)
	at com.bigdata.journal.jini.ha.TestHA3JournalServer.testQuorumBreaksABC_failLeader(TestHA3JournalServer.java:1329)
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase$9.run(AbstractHA3JournalServerTestCase.java:2031)
	at com.bigdata.io.TestCase3.assertCondition(TestCase3.java:223)
	... 22 more
Caused by: java.util.concurrent.TimeoutException
	at com.bigdata.quorum.AbstractQuorum.awaitQuorum(AbstractQuorum.java:1159)
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase$9.run(AbstractHA3JournalServerTestCase.java:2022)
	... 23 more

- testABCMultiLoadFollowerReadsLargeLoad: Failed once (trace
  below). 2nd time passed.

java.util.concurrent.ExecutionException: com.bigdata.quorum.QuorumException: Quorum not met on token: expected 0, actual=1
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
	at java.util.concurrent.FutureTask.get(FutureTask.java:111)
	at com.bigdata.journal.jini.ha.TestHA3JournalServer.doABCMultiLoadFollowerReads2(TestHA3JournalServer.java:2015)
	at com.bigdata.journal.jini.ha.TestHA3JournalServer.testABCMultiLoadFollowerReadsLargeLoad(TestHA3JournalServer.java:1901)
Caused by: com.bigdata.quorum.QuorumException: Quorum not met on token: expected 0, actual=1
	at com.bigdata.quorum.AbstractQuorum.assertQuorum(AbstractQuorum.java:1065)
	at com.bigdata.quorum.AbstractQuorumClient.getLeader(AbstractQuorumClient.java:106)
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase$LargeLoadTask.call(AbstractHA3JournalServerTestCase.java:2312)
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase$LargeLoadTask.call(AbstractHA3JournalServerTestCase.java:1)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
}}}

@see https://sourceforge.net/apps/trac/bigdata/ticket/530 (Journal HA)

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7209 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Add directWrite state variable to ensure that flush() writes all buffers to backing store/ha pipeline, including the compacting buffer.

Ticket #674

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/READ_CACHE@7210 8f7bc0f5-282e-0410-95e3-8d296e9bb460