Ticket 47520 - Fix various issues with logconv.pl

Bug Description:  When testing an access log that contains "internal" connections/ops, many
                  warning messages are created.  Other warnings are generated because the
                  DB_File arrays need to have thier elements checked before comparing to other
                  strings.

Fix Description:  Improve value checking.  Add support for connection numbers that are words (Internal),
                  and negative values for op ids(-1).

https://fedorahosted.org/389/ticket/47520

Reviewed by: richm(Thanks!)
Ticket 47509 - CLEANALLRUV doesnt run across all replicas

Bug Description:  If one of the replicas in the environment is older and does not support
                  cleanallruv, then this would cause the cleanallruv task to never complete.

Fix Description:  Several issues are addressed with this fix:

	- Corrected error handling of replicas that do not support cleanallruv(prevent task hanging)

	- Created helper function to get local maxcsn:  replica_cleanallruv_get_local_maxcsn().
          This code was taken from the extended op "multimaster_extop_cleanruv_get_maxcsn".

	- Improved replica_cleanallruv_find_maxcsn() to also include the local maxcsn, and not just the remote replicas.

	- Improved abort task processing.  Abort task now waits for the cleanallruv task to be aborted before continuing.

	- Improved task configuration cleanup.  Removed places where the config attribute value was manually created.
          Now we search for the exact value to delete.

https://fedorahosted.org/389/ticket/47509

Reviewed by: richm(Thanks!!)
Ticket 47509 - Cleanallruv jenkins error

Fixed invalid array element that was out of bounds.

https://fedorahosted.org/389/ticket/47509
coverity 11956 - Dereference after null check

Fix description: The NULL check is if either (remote_subtree or
local_subtree is NULL) and (subtree_pairs is NULL), then it bails.
The location that Coverity reported is changed to "strictly"
subtree_pairs is NULL.  That is, local_subtree cannot be NULL.

I applied the fix to the similar places.

Reviewed by Rich (Thank you!!)
Ticket #47492 - PassSync removes User must change password flag on the Windows side

Bug description: Windows Sync sends password modify even if it is
from PassSync originated on AD.  The modify updates the pwdLastSet
attribute value to non-zero value.  The value 0 indicates the pass-
word must change at next logon on AD.

Fix description: Before sending the password modify, check whether
the current pwdLastSet value is 0 or not.  If it is 0 (means the
password must change), reset pwdLastSet value to 0 along with the
password modify.  This operation replaces the password on AD, but
the password still must change at next logon.

Note: If "password must change at next logon" on the both DS and AD,
the password needs to be changed by the user on the both servers to
enable it on each.

https://fedorahosted.org/389/ticket/47492

Reviewed by Rich (Thank you!!)
Ticket 47513 - tmpfiles.d references /var/lock when they should reference /run/lock

Bug Description:  Due to a possible race condition, the server should be using /run for
                  /var/lock and /var/run

Fix Description:  To be backwards compatible, we first check if /run exists(as it does not exist in rhel 6.x), then set
                  the lock directory to be /run/lock & /var/run to /run, and only write this to our tmpfiles.d configuration.

https://fedorahosted.org/389/ticket/47513

Reviewed by: richm(Thanks!)
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: master
Fix Description: Added a new attribute nsIndexIDListScanLimit to nsIndex
 nsIndexIDListScanLimit: limit=NNN [type=eq[,sub,...]] [flags=ADD[,XXX..]] [values=val[,val...]]
* The limit is the idlistscanlimit to apply.  The value can be -1 (unlimited), 0
(do not use the index at all), or a number to use for the idlistscanlimit.
* The type is a comma-delimited list of index types (eq,sub,etc.) to which the
limit applies.  The index type must be one of the index types configured for the index (you can't configure an id list for type=sub if you do not already have
a substring index configured for the attribute).
* The flags are a comma-delimited list of additional criteria which must match.
** flags ADD - only apply the limit when the index is used in an AND (&) filter
The values are a comma-delimited list of values which must match in the search
filter in order for the limit to be applied.  Since the matches are done one
at a time (we evaluate one filter component at a time), the values will match
if any of the values match.
* The values must be used with only one type at a time.  If values are specified,
type must be specified, and type must be a type that deals with values, such
as eq or sub.  There must be only one type specified - you can't specify values
if you use type=eq,pres or otherwise specify more than one type.  The values
must correspond to the index type (eq, sub), and must correspond to the syntax
of the attribute to which the index is applied - that is, if you have
attribute uidNumber (integer) and it is indexed for eq, you can't specify
type=eq values=abc because "abc" is not integer syntax.
If the values contain spaces, commas, nulls, other values which require
escapes, the LDAP filter escape syntax should be used - backslash '\' followed
by the 2 hex digit code for the character.
The values are processed as if they were filter values - so for "sub" values,
values like "values=*sm*ith*" will be processed as if they were values in a
substring search filter like (sn=*sm*ith*)
* nsIndexIDListScanLimit is multi-valued.  If a search matches more than one
nsIndexIDListScanLimit, the rules are applied in priority order.
The priority is as follows, from highest to lowest:
 * * match type, flags, value
 * * match type, value
 * * match type, flags
 * * match type
 * * match flags
 * For example, if you have
 * dn: cn=objectclass,...
 * objectclass: nsIndex
 * nsIndexType: eq
 * nsIndexIDListScanLimit: limit=0 type=eq flags=AND value=inetOrgPerson
 * nsIndexIDListScanLimit: limit=1 type=eq value=inetOrgPerson
 * nsIndexIDListScanLimit: limit=2 type=eq flags=AND
 * nsIndexIDListScanLimit: limit=3 type=eq
 * nsIndexIDListScanLimit: limit=4 flags=AND
 * nsIndexIDListScanLimit: limit=5
 * If the search filter is (&(objectclass=inetOrgPerson)(uid=foo)) then the limit=0 because all
 *  3 of type, flags, and value match
 * If the search filter is (objectclass=inetOrgPerson) then the limit=1 because type and value match
 *  but flag does not
 * If the search filter is (&(objectclass=posixAccount)(uid=foo)) the the limit=2 because type and
 *  flags match
 * If the search filter is (objectclass=posixAccount) then the limit=3 because only the type matches
 * If the search filter is (&(objectclass=*account*)(objectclass=*)) then the limit=4 because only
 *  flags match but not the types (sub and pres)
 * If the search filter is (objectclass=*account*) then the limit=5 because only the attribute matches
 *  but none of flags, type, or value matches
To add in testing/debugging, the LDAP_DEBUG_BACKLDBM log level is used to
print information about searches which exceed the idlistscanlimit.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: yes - document new attribute
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: master
Fix Description: Fix compiler warning - use bv.bv_val to print syntax
errors in substring filter.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
Ticket 601 - multi master replication allows schema violation

Bug Description:  If a required attribute has two values, it is possible
			to delete them concurrently on different masters.
			The delete of a single value is valid, but after
			the deletes are replicated all values are gone.

Fix Description:   In the state of update resoultion it is too late to reject
			any of the deletes (and it would require schema check
			in the update resolution process), they have been acked
			to the client.
			What could an dshould be done is to detect this situation
			log an error and flag the entry with a replConflict.
			To do this schema checking for mods has to be done for replicated
			operations and the slapi_enty_schema_check function needs
			to be extended.

https://fedorahosted.org/389/ticket/601

Reviewed by: ?
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: master
Fix Description: This patch broke replication.  The problem is that we cannot
compare the allidslimit to 0 unless the value is explicitly set by
index_get_allids.  The fix is to only return ALLIDS if the allidslimit was
explicitly set to 0 by index_get_allids.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
Ticket 47528 - 389-ds-base built with mozldap can crash from invalid free

Bug Description:  The issue is that the slapi_escape_filter_value() returned string gets
                  freed by the caller. When using mozldap, this function can return the
                  original filter pointer, which can lead to a double free.

Fix Description:  Return a copy of the filter str so it can be safely freed.  Also free
                  the buf if it's not being returned.

https://fedorahosted.org/389/ticket/47528

Reviewed by: nhosoi(Thanks!)
Ticket #47523 - Set up replcation/agreement before initializing the sub suffix, the sub suffix is not found by ldapsearch

Bug description: If a replication is configured against a backend
before initializing the backend with a suffix entry, an RUV entry
is inserted first with the entryid 1. The RUV entry's entryrdn is
added to the entryrdn index with a suffix entry which is a parent
entry of the RUV entry having a temporary entryid 0, which was to
be replaced with the real entryid when the real suffix entry is
added.  But the replacement code was not executed.

Fix description: When a real suffix is added to the entryrdn index,
it returns DB_KEYEXIST, which used to be ignored by resetting 0
(== SUCCESS).  This patch returns DB_KEYEXIST to the caller and let
_entryrdn_insert_key use the info to replace the temporary entryid
with the real one.  The error code is ignored by the other callers.

https://fedorahosted.org/389/ticket/47523

Reviewed by nkinder (Thanks!).
Ticket #47501 logconv.pl uses /var/tmp for BDB temp files

https://fedorahosted.org/389/ticket/47501
Reviewed by: mreynolds (Thanks!)
Branch: master
Fix Description: Tied arrays with DB_RECNO require a backing text file.  By
default this goes in /var/tmp.  This can be controlled with RECNOINFO bfname.
However, the use of arrays was killing performance.  This patch replaces
all arrays with tied hashes.  In addition, this patch fixes the notes=U
handling to also handle notes=U,P and notes=A,P
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
Ticket 47388 - RFE to implement RFC4533 -ver2

Bug Description:  Request to implement RFC4533 in ds 389

Fix Description:   This fix impements the content synchronization as
			a directory server plugin, using the retro changelog
			to provide synchronization with cookies.
			The design is presented here:
			http://port389.org/wiki/Content_synchronization_plugin

https://fedorahosted.org/389/ticket/47388

Reviewed by:  nkinder, Thanks
Ticket 47510 - 389-ds-base does not compile against MozLDAP libraries

Used #ifdef's to remove openldap specific API calls.

Thanks to mvocu for providing the initial patch.

https://fedorahosted.org/389/ticket/47510

Reviewed by: rmeggins(Thanks!)
Ticket 47532 - 1.3.1 with mozldap crashes in new operation work_q code

Bug Description:  The changes from ticket 513 seem to trigger two crashes when using mozldap.

Fix Description:  When freeing a operation/ber we need to check if the ber is NULL, and
                  when getting a operation from the stack and stack operation is NULL,
                  we need to create a new operaton, and not try to init the empty one.

https://fedorahosted.org/389/ticket/47532

Reviewed by: rmeggins(Thanks!)
Ticket 47533 logconv: some stats do not work across server restarts

https://fedorahosted.org/389/ticket/47533
Reviewed by: mreynolds (Thanks!)
Branch: master
Fix Description: Most of the stat hashes use connid,opid as the hash key.
The connid,opid are not unique because the server recycles connection ids
when the server restarts.  The solution is to use a triplet of
$serverRestartCount,$connid,$opid
as the hash key.  This uniquely identifies a particular log event over
restarts.  There is also some additional cleanup that needs to be done when
the server is reset - returning fds, marking connections as closed, etc.
Also fixed some gmtime handling - we were doing too many conversions.
This fix adds another command line option - -U - for a summary of search
filters which are unindexed or have unindexed components.  The full detailed
report is just too much data.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
Ticket #47534 - RUV tombstone search with scope "one" doesn`t work

Bug description: slapi_sdn_scope_test_ext examines the given dn
satisfies the scope against the base dn.  Onelevel case for the
tombstone DN ignores the nsuniqueid leaf rdn before checking the
scope which should not be done for the RUV tombstone.

Fix description: This patch correctly handles the RUV tombstone
as an exception.

https://fedorahosted.org/389/ticket/47534

Reviewed by rmeggins (Thanks, Rich!)
(cherry picked from commit fbece32d94515bc1ca3889c3d7c0691bbeec3ab0)
Ticket 47510 - Repl Sync does not compile against MozLDAP libraries

Repl Sync uses openldap extensively, so we had to bring over many defines.  Also
fixed some other warnings.

https://fedorahosted.org/389/ticket/47510

Reviewed by: rmeggins(Thanks!)
Ticket 47510 - remove unnecessary typedef

https://fedorahosted.org/389/ticket/47510