OPENDJ-1206 (CR-4368) Create a new ReplicationBackend/ChangelogBackend to support cn=changelog

Preparation work: ChangeNotificationListener interface only cover a subset of its DirectoryServerPlugin superclass capabilities, so it is redundant.
So I removed ChangeNotificationListener to ease moving persistent searches to the backend.
The two ChangeNotificationListener implementations are now subclasses of InternalDirectoryServerPlugin.
Updated all code after removing ChangeNotificationListener.



ChangeNotificationListener.java: REMOVED



AuthenticatedUsers.java:
Now extends InternalDirectoryServerPlugin instead of implementing ChangeNotificationListener.
Extracted method getNewAuthNDN() + toDN().

CryptoManagerSync.java:
Now extends InternalDirectoryServerPlugin instead of implementing ChangeNotificationListener.
Extracted method getNewAuthNDN() + toDN().



DirectoryServer.java:
Removed references to removed type ChangeNotificationListener.
Be careful with when AuthenticatedUsers is initialized.
Added initializeAuthenticatedUsers().

DirectoryConfig.java, LocalBackend*Operation.java:
Removed references to removed type ChangeNotificationListener.

EncodePassword.java:
In initializeServerComponents(), called DirectoryServer.initializeAuthenticatedUsers() to initialize DirectoryServer.authenticatedUsers and avoid the NPE.
Extracted method listPasswordStorageSchemes().
Code cleanup.


CollectionUtils.java: ADDED

TestChangeNotificationListener.java:
Now extends InternalDirectoryServerPlugin instead of implementing ChangeNotificationListener.

AddOperationTestCase.java:
Extracted methods newRawAttributes(), byteStrings(), addSuccess(), addDisconnect(), bind(), writeAddRequest() + used CollectionUtils.
Consequence of the changes to TestChangeNotificationListener.

DeleteOperationTestCase.java:
Consequence of the changes to TestChangeNotificationListener.

ModifyOperationTestCase.java:
Extracted methods newModifications(), newRawModifications() + used CollectionUtils.
Consequence of the changes to TestChangeNotificationListener.


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11025 41b1ffd8-f28e-4786-ab96-9950f0a78031
OPENDJ-1389 – Add support for replication changelog DB rotation
 
Implemented a log based on multiple log files 

[Note: real merge of all changelog.file package content to be done in one shot in
 a future commit]

* Add Log class that manage a log as a set of log files:
** it contains at least one log file, the head log file, 
     where new records are appended
** it contains from zero to multiple read-only log files, 
    issued from rotation of the head log file when it reaches
    a given size
* Update LogFile class to act as part of a Log.
* Add purging feature
* Update other classes from file package with minor changes
* Add unit tests for new Log class and purging feature

* Update MeteredStream class to use it in changelog
* Add new interface RotatableLogFile (to be used later by changelog)
* Update Policy classes to use this new interface

git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11026 41b1ffd8-f28e-4786-ab96-9950f0a78031
Code cleanup


SearchOperation.java, SearchOperationBasis.java, SearchOperationWrapper.java:
Changed entriesSent and referencesSent from int to AtomicInteger.
Removed unused fields eclRequestControl, processingStartTime and processingStopTime.
Simplified constructors by initializing the fields during where they are declared.
Extracted method getACIHandler().
Removed incrementEntriesSent() and incrementReferencesSent().
Code cleanup.

Backend.java:
Removed unused code.
Removed ctor, initialized fields where they are declared.

WorkflowTopology.java:
Added toString().
Code cleanup.


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11027 41b1ffd8-f28e-4786-ab96-9950f0a78031
OperationWrapper.java, AbstractOperation.java, Operation.java, PluginOperation.java:
In getAttachment(), removeAttachment() and setAttachment(), changed the return type to a generic type. This simplifies client code by removing the need to cast the returned value.


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11029 41b1ffd8-f28e-4786-ab96-9950f0a78031
OPENDJ-1467 :  File Based Changelog must support replicas temporarily leaving the topology

[Note: real merge of all changelog.file package content to be done in one shot in
 a future commit]
 
* File-based changelog
 Store the offline CSN when a replica goes offline
 Read the offline CSN if present to build the changelog state at startup
 
 Update ReplicationEnvironment.java to manage read and write from/to storage
 Update ReplicationEnvironmentTest.java with more unit tests
 Update replication.properties with new messages
 
* File and JE based changelog
 Remove the offline CSN if present when receiving an heartbeat or an
  update message
  
 Update FileChangelogDB.java, JEChangelogDB.java, ReplicationDBEnv.java to
 manage online replica notification
 
* Other minor updates : renaming, comments



git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11030 41b1ffd8-f28e-4786-ab96-9950f0a78031
OPENDJ-1483: On Windows, problem with changelog's file rotation when running 
  replication topology with file-based changelog
OPENDJ-1487: File based changelog : cursors opened when clearing the log

[Note: real merge of all changelog.file package content to be done in one shot in
 a future commit]

Log.java :
* Ensure cursors opened on log are temporarily disabled before rotating
 the head (to avoid renaming the file while reader are opened on it)
* Allow cursors to be opened on log when performing a clear on the log, 
 turning them into empty cursors. 


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11031 41b1ffd8-f28e-4786-ab96-9950f0a78031
OPENDJ-1472 : File based changelog : optimize random seek in each log file
CR-3727

[Note: real merge of all changelog.file package content to be done in one shot in
 a future commit]

Implements read and write with blocks, to enable binary search.

* New BlockLogReader class: read records sequentially or using
 binary search provided a key
 
* New BlockLogWriter class: write records by blocks

* New BlockLogReaderWriterTest class : tests read/write with blocks

* Update LogFile : delegate responsibility for read/write to BlockLogXXX

* Add method ByteStringBuilder#append(DataInput, int) to avoid 
 byte array copy

* Minor changes in other files


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11032 41b1ffd8-f28e-4786-ab96-9950f0a78031
OPENDJ-1449 : File-based changelog should handle partially written record left over from a previous failure
CR-3768

[Merge r10816 for Nicolas Capponi]
[Note: real merge of all changelog.file package content to be done in one shot in
 a future commit]

* LogFile.java
** Add a check of file validity in constructor when log file is write-enabled
** Recover the file if it is corrupted (partially written record) by truncating it

* BlockLogReader.java
** Add method checkLogIsValid() to check validity of file
 
* replication.properties
** Add new messages related to check and recovery

* LogFileTest.java
** Add tests for recovery after log file corruption

* Minor changes in other files


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11034 41b1ffd8-f28e-4786-ab96-9950f0a78031
Forward port OPENDJ-1565 CR-4564: Optionally bind to a local address

git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11036 41b1ffd8-f28e-4786-ab96-9950f0a78031
OPENDJ-1471 File based changelog : improve cursor behavior
CR-3920

[Note: real merge of all changelog.file package content to be done in one shot in
 a future commit]

Third and last step : all cursors related to ChangeNumberindexDB now 
 behave like java.sql.ResultSet API. 

* FileChangeNumberIndexDBCursor, JEChangeNumberIndexDBCursor: 
 now behave like ResultSet API

* ECLServerHandler : adapt to changes of underlying cursor 
 on ChangeNumberIndexDB

* FileChangeNumberIndexDBTest, JEChangeNumberIndexDBTest: 
 adapt to changes on cursor

 * ChangeNumberIndexDB : javadoc fix
 
 * ReplicationEnvironment : change default log file size to 10M

git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11038 41b1ffd8-f28e-4786-ab96-9950f0a78031
Checkpoint commit for OPENDJ-1206 : Create a new ReplicationBackend/ChangelogBackend 
 to support cn=changelog
CR-4036

Preparatory work :  change implementation of virtual attribute providers related to external
 changelog to be independant of ECL workflow

*  ReplicationServer class:
** remove duplication of virtual attribute providers creation (for registering
    and deregistering)
** create virtual attribute providers by injecting the replication server instance

* FirstChangeNumberVirtualAttributeProvider, LastChangeNumberVirtualAttributeProvider, 
  LastCookieVirtualProvider classes:
** inject ReplicationServer in constructor in order to use it in getValues() method
** remove ConfigurationChangeListener implementation because no config change 
 can happen

* ChangelogBaseDNVirtualAttributeProvider class:
** remove ConfigurationChangeListener implementation because no config change 
 can happen

git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11039 41b1ffd8-f28e-4786-ab96-9950f0a78031
Checkpoint commit for OPENDJ-1206 : Create a new ReplicationBackend/ChangelogBackend 
 to support cn=changelog
CR-4052

Preparatory work for the changelog backend: request control handling

* LocalBackendSearchOperation: add new internal method backendSupporsControl()


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11040 41b1ffd8-f28e-4786-ab96-9950f0a78031
Checkpoint commit for OPENDJ-1206 : Create a new ReplicationBackend/ChangelogBackend 
 to support cn=changelog
CR-4083

[Note: real merge of all changelog.file package content and ChangelogBackend to be done
 in one shot in a future commit]

Implementation of core features of the changelog backend:
* Initialization and finalization of the backend
* Search of the changelog, in cookie mode (with a cookie control) and
  draft compat mode (by change number)

Code changes:
* ChangelogBackend.java: 
   - implementation of search(), hasSubordinates(),  numSubordinates(),
     and getEntryCount() methods 

* ReplicationServer.java:
   - new dependency on ChangelogBackend and ECLEnabledDomainPredicate
   - new constructor with ECLEnabledDomainPredicate argument
   - new methods enableExternalChangeLog() and shutdownExternalChangelog()
     as future replacement of enabledECL() and shutdownECL()
   - new method getDomainDNs(Set<DN>) for retrieval of domain DNs but an 
     excluded set of dns
   - new method validateServerState(MultiDomainServerState, Set<DN>) for 
     checking coherency of given state with the replication server

* ReplicationDomainDB.java: 
   - new method getCursorFrom(MultiDomainServerState, PositionStrategy, Set<DN>)
     that exclude a given set of domain DNs from the cursor obtained

* FileChangelogDB.java, JEChangelogDB.java:
   - implementation of new method 
     getCursorFrom(MultiDomainServerState, PositionStrategy, Set<DN>)

* ECLEnabledDomainPredicate.java, ECLMultiDomainDBCursor.java:
  - update visibility to public in order to use these classes in ChangelogBackend

* ChangelogBackedTestCase.java:
   - test of ChangelogBackend class, built from ExternalChangeLogTest.java,
     with lots of renaming, refactoring, cleaning compared to original class
   - majority of tests are disabled until the changelog backend is branched into
     code (as these tests require a running server)

* MonitorTest.java:
  - update creation of  ReplicationServer class to use a custom 
    ECLEnabledDomainPredicate
  
 * replication.properties:
   - add messages for changelog backend


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11042 41b1ffd8-f28e-4786-ab96-9950f0a78031
Checkpoint commit for OPENDJ-1206: Create a new ReplicationBackend/ChangelogBackend 
 to support cn=changelog

Ignore unused domains when validating a server state for a changelog search. This was
the prior behavior withe ECL* implementation.

* ReplicationServer.java:
  Update validateServerState() method to ignore unused domain

git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11043 41b1ffd8-f28e-4786-ab96-9950f0a78031
Checkpoint commit for OPENDJ-1206 Create a new ReplicationBackend/ChangelogBackend 
 to support cn=changelog
CR-4439

Implementation of an additional validation of provided cookie when searching changelog
in cookie mode. 

Check for each state of the provided cookie that the csn is not older than the oldest CSN
available in each domain. If this is not the case, then an error 
ERR_RESYNC_REQUIRED_TOO_OLD_DOMAIN_IN_PROVIDED_COOKIE is returned. 

This check was done previously in the ECLServerHandler class but was not ported when
writing ChangelogBackend.  It also means I'm re-adding some methods deleted by the 
previous big clean done by Jean-Noel. 

Changes:
* ReplicationServer.java : 
  - add the check to validateServerState() method
  - refactor validateServerState() method for better readability
* ReplicationServerDomain.java : 
  - re-add getOldestState() method
* ReplicationDomainDB. java : 
  - re-add getDomainOldestCSNs(DN) method
* FileChangelogDB.java, JEChangelogDB.java : 
  - re-add implementation of getDomainOldestCSNs(DN) method


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11044 41b1ffd8-f28e-4786-ab96-9950f0a78031
OPENDJ-1444 CR-4537 Remove previous cookie from storage of ChangeNumberIndexDB

[Note: real merge of all changelog.file package content and ChangelogBackend to be done 
 in one shot in a future commit]

* Implement a new matching strategy for cursors : LESS_THAN_OR_EQUAL_TO_KEY
   for both je and file-based implementations

* Replace the previous cookie by the usage of an ECLMultiDomainDBCursor generated 
   with the medium consistency CSN as start point and the 
   LESS_THAN_OR_EQUAL_TO_KEY strategy
   in classes ChangeNumberIndexer and ChangelogBackend

* Remove storage of the previous cookie in the log for both je and file-based
   implementations


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11045 41b1ffd8-f28e-4786-ab96-9950f0a78031
Rename type parameter of CompositeDBCursor class from Data to T

git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11046 41b1ffd8-f28e-4786-ab96-9950f0a78031
Actual merge of complete changelog.file package which contains implementation
of file-based changelog

git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11048 41b1ffd8-f28e-4786-ab96-9950f0a78031
Actual merge of ChangelogBackend class

git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11049 41b1ffd8-f28e-4786-ab96-9950f0a78031
OPENDJ-1206 (CR-4393) Create a new ReplicationBackend/ChangelogBackend to support cn=changelog

Moved persistent searches from workflow elements to the backends.
Made persistent searches work for the ChangelogBackend.
Ensured the base changelog entry is always returned before any changelog entry, even with persistent searches. For this, used SearchOperation attachments to pass information from the "initial search" phase to the "persistent search" phase.
Added ChangelogBackend.notifyEntryAdded() and called it directly from the (JE|File)ChangelogDB and the ChangeNumberIndexer.

FIXME: prevent concurrent execution of initial vs. persistent search phases to avoid sending duplicates.
How do other backends deal with such issue? Apparently they don not deal with it.
TODO: verify changelog read privilege for persistent searches




Backend.java
Added persistentSearches field + registerPersistentSearch() and getPersistentSearches().

ChangelogBackend.java:
Replaced baseChangelogDN with CHANGELOG_BASE_DN and used it throughout + simplified ctor.
Added notifyEntryAdded(), getPersistentSearches(), isPersistentSearch(), isCookieBased(), getNewestCookie(), getChangelogDB(), getInstance(), NumSubordinatesSearchOperation.setAttachment(), TODO JNR
Overrode Backend.registerPersistentSearch().
Extracted method getExcludedDomains().
Renamed SearchParams.multiDomainServerState to cookie.
Removed unused SearchParams.operationId.
Added inner class EntrySender and moved methods to it: matchBaseAndScopeAndFilter(), sendBaseChangelogEntry(), buildBaseChangelogEntry(),
In searchFromCookie(), searchFromChangeNumber() and sendEntryForUpdateMessage(), simplified the code by using EntrySender.
Implemented finalizeBackend().

*Backend.java:
Called super.finalizeBackend()

ChangelogBackendTestCase.java:
Enabled as many tests as possible.



PersistentSearch.java
Added changesOnly field + isChangesOnly() + modified ctor to set it.

ReplicationServer.java:
Replaced old code enabling External Changelog (via workflow element) with new code (with ChangelogBackend).



FileChangelogDB.java, ChangeNumberIndexer.java, JEChangelogDB.java:
Called new ChangelogBackend.notifyEntryAdded().

ChangeNumberIndexerTest.java
Consequence of the change to ChangeNumberIndexer.



LDAPReplicationDomain.java:
Removed old code enabling External Changelog.
Code cleanup.

ExternalChangeLogTest.java:
Disabled tests that do not pass anymore.
Removed old code enabling External Changelog.
Code cleanup.



ECLWorkflowElement.java, LocalBackendWorkflowElement.java:
Removed persistentSearches field + registerPersistentSearch() and getPersistentSearches().

LocalBackendAddOperation.java, LocalBackendDeleteOperation.java, LocalBackendModifyDNOperation.java:
Consequence of moving persistent searches from workflow elements to the backends.

LocalBackendModifyOperation.java:
Consequence of moving persistent searches from workflow elements to the backends.
In performAdditionalPasswordChangedProcessing(), simplified code because we are adding to a Set.
Code cleanup.

LocalBackendSearchOperation.java:
Consequence of moving persistent searches from workflow elements to the backends.
Transformed processSearch field into a local variable.
Code cleanup.

ECLServerHandler.java, ECLServerWriter.java, ECLSearchOperation.java:
Adapted the code to use ChangelogBackend.


git-svn-id: https://svn.forgerock.org/opendj/branches/opendj3-server-dev@11050 41b1ffd8-f28e-4786-ab96-9950f0a78031