inline comments and final fields and args.

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7213 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Patched RemoteRepositoryManager per [1].

This ticket is still open.  There are similar problems that have been identified in RemoteRepository. That class needs to be surveyed and all such issues identified.

Once the changes have been applied, they will need to be merged from branches/BIGDATA_RELEASE_1_2_0 into branches/READ_CACHE.

[1] https://sourceforge.net/apps/trac/bigdata/ticket/696 (Incorrect HttpEntity consuming in RemoteRepositoryManager)

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7215 8f7bc0f5-282e-0410-95e3-8d296e9bb460
fixed a concurrent modification exception

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7216 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Test commit - corect a comment.


git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7218 8f7bc0f5-282e-0410-95e3-8d296e9bb460
New Helper class to make it easier to write AST tests, and keep the comments in sync with the test; since the test and the comments are the same.

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7220 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Copied test NSI01 to new Helper

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7221 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Added test for property path union as in trac684

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7222 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Added two further tests from trac 684 entry

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7223 8f7bc0f5-282e-0410-95e3-8d296e9bb460
small tidy up

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7224 8f7bc0f5-282e-0410-95e3-8d296e9bb460
renamed various methods to be more conformant with Java naming conventions

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7225 8f7bc0f5-282e-0410-95e3-8d296e9bb460
renamed two new failing tests - in order to ignore them, before passing to Mike

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7226 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Bug fix for [1]

[1] https://sourceforge.net/apps/trac/bigdata/ticket/706 (MultiSourceSequentialCloseableIterator.nextSource() can throw NPE)


Commit is against the 1.2.x maintenance branch.

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7233 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Bug fix for [1]

Rather, I would say that putIfAbsent() is simply not checking to see whether the QueryEngine has been shutdown.
{{{
    protected AbstractRunningQuery putIfAbsent(final UUID queryId,
            final AbstractRunningQuery runningQuery) {

        if (queryId == null)
            throw new IllegalArgumentException();

        if (runningQuery == null)
            throw new IllegalArgumentException();

        // First, check [runningQueries] w/o acquiring a lock.
        {
            final AbstractRunningQuery tmp = runningQueries.get(queryId);

            if (tmp != null) {
        
                // Found existing query.
                return tmp;
                
            }
            
        }

        /*
         * A lock is used to address a race condition here with the concurrent
         * registration and halt of a query.
         */

        lock.lock();
        
        try {
            
            // Test for a recently terminated query.
            final Future<Void> doneQueryFuture = doneQueries.get(queryId);

            if (doneQueryFuture != null) {

                // Throw out an appropriate exception for a halted query.
                handleDoneQuery(queryId, doneQueryFuture);

                // Should never get here.
                throw new AssertionError();
                
            }

            // Test again for an active query while holding the lock.
            final AbstractRunningQuery tmp = runningQueries.putIfAbsent(queryId,
                    runningQuery);
            
            if (tmp != null) {
                
                // Another thread won the race.
                return tmp;
                
            }

            // Query was newly registered. <==  CHANGE IS APPLIED HERE
            return runningQuery;
            
        } finally {

            lock.unlock();
            
        }

    }
}}}

The proposed change is below. The position of this change in the method is marked above.
{{{
            /*
             * Query was newly registered.
             */
            try {
                
                // Verify QueryEngine is running.
                assertRunning();
                
            } catch (IllegalStateException ex) {
                
                /**
                 * The query engine either is not initialized or was shutdown
                 * concurrent with adding the new query to the running query
                 * table. We yank the query out of the running query table in
                 * order to have no net effect and then throw out the exception
                 * indicating that the QueryEngine has been shutdown.
                 * 
                 * @see <a
                 *      href="https://sourceforge.net/apps/trac/bigdata/ticket/705">
                 *      Race condition in QueryEngine.putIfAbsent() </a>
                 */

                runningQueries.remove(queryId, runningQuery);

                throw ex;

            }        
}}}

The BOP, QueryEngine, and TestBigdataSailWithQuads test suites are unchanged by this modification.

[1] https://sourceforge.net/apps/trac/bigdata/ticket/705 (Race condition in QueryEngine.putIfAbsent())

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7234 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Added two unit tests and an optional stress test to verify that BlockingBuffer.close() correctly wakes up a producer that is blocked on add().

@see https://sourceforge.net/apps/trac/bigdata/ticket/707 (BlockingBuffer.close() does not unblock threads)

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7235 8f7bc0f5-282e-0410-95e3-8d296e9bb460
The failure to wake the producer when you call buffer.iterator().close() is because you are not calling buffer.setFuture() with the Future of the producer in your test.  I have committed a test which does this and shows that the producer is correctly awoken (the test observes a CancellationException for the Future).

Per the ticket, please verify under a debugger that the iterator() and/or buffer were closed for your stack traces.  I suspect that they were not closed and that the traces exist because the query was not correctly terminated.  It is also possible that setFuture() was not called on the buffer or perhaps that a race condition existing for setting the Future on the buffer.  However, I need more information to diagnose this furthre.

See https://sourceforge.net/apps/trac/bigdata/ticket/707 (BlockingBuffer.close() does not unblock threads)

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7236 8f7bc0f5-282e-0410-95e3-8d296e9bb460
Backport of fix for [1] to the 1.2.x maintenance branch. 

Note: This fix was applied to the READ_CACHE branch in r7203.

[1] https://sourceforge.net/apps/trac/bigdata/ticket/646 (River not compatible with newer 1.6.0 and 1.7.0 JVMs)

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7240 8f7bc0f5-282e-0410-95e3-8d296e9bb460
I have identified several cases where IBlockingBuffer.setFuture() was not being invoked until after the Future had already begun to execute. This could result in the producer (the Future is for the producer) not being cancelled if the consumer was closed before the setFuture() was invoked.  I have updated the code to use a FutureTask to wrap the computation, use setFuture(ft) for that FutureTask, and only then submit the FutureTask for evaluation. This ensures that the Future of the producer is set before the consumer starts to run.  Changes were made to the following files:

{{{
- AccessPath#asynchronousIterator()
- AbstractClientIndexView#newWriteBuffer()
- ClientIndexView#newWriteBuffer()
- ClientIndexView#parallelRangeIterator()
- ClientAsynchronousIterator#start()
- MappedTaskMaster#newResourceBuffer()
- DGExpander.InnerIterator#312
- TripleStoreUtility#notFoundInTarget()
}}} 

In addition, the following unit tests had the bad pattern and were fixed:
{{{
- TestBlockingBuffer
- TestFileSystemScanner
- TestMasterTask
- TestMasterTaskWithTimeout
- TestMasterTaskWithErrors
- TestMasterTaskWithRedirect
- TestMasterTaskWithSplits (note: this test is no in CI - never finished)
}}}

There were two places in the code where I could not apply this pattern because the consumer was started in a different lexical scope.  For these two cases, I wrapped the code to ensure that the Future was cancelled if it there was an exit by a code path that did not cause setFuture() to be invoked.  
ProgramTask#362
{{{
            try {
                // run the task.
                future = queryTask.submit();

                // set the future on the BlockingBuffer.
                buffer.setFuture(future);
            } finally {
                if (future != null && buffer.getFuture() == null) {
                    // Future exists but not set on BlockingBuffer.
                    future.cancel(true/* mayInterruptIfRunning */);
                }
            }
}}}

AbstractMasterTask#905:
{{{
                Future<? extends AbstractSubtaskStats> future = null;
                try {
                    // assign a worker thread to the sink.
                    future = submitSubtask(sink);
                    // set Future (can now be cancelled)
                    out.setFuture(future);
                } finally {
                    if (future != null && buffer.getFuture() == null) {
                        // Future exists but not set on BlockingBuffer.
                        future.cancel(true/* mayInterruptIfRunning */);
                    }
                }
}}}

See https://sourceforge.net/apps/trac/bigdata/ticket/707 (BlockingBuffer.close() leaks threads)

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7242 8f7bc0f5-282e-0410-95e3-8d296e9bb460
fixed 712 - SELECT ?x { OPTIONAL { FILTER (false) }}, including test cases

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7243 8f7bc0f5-282e-0410-95e3-8d296e9bb460
checking in Jeremy's fix for ticket 684

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7244 8f7bc0f5-282e-0410-95e3-8d296e9bb460
checking in Jeremy's fix for ticket 684

git-svn-id: https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0@7245 8f7bc0f5-282e-0410-95e3-8d296e9bb460