Ticket 47509 - CLEANALLRUV doesnt run across all replicas

Bug Description:  If one of the replicas in the environment is older and does not support
                  cleanallruv, then this would cause the cleanallruv task to never complete.

Fix Description:  Several issues are addressed with this fix:

	- Corrected error handling of replicas that do not support cleanallruv(prevent task hanging)

	- Created helper function to get local maxcsn:  replica_cleanallruv_get_local_maxcsn().
          This code was taken from the extended op "multimaster_extop_cleanruv_get_maxcsn".

	- Improved replica_cleanallruv_find_maxcsn() to also include the local maxcsn, and not just the remote replicas.

	- Improved abort task processing.  Abort task now waits for the cleanallruv task to be aborted before continuing.

	- Improved task configuration cleanup.  Removed places where the config attribute value was manually created.
          Now we search for the exact value to delete.

https://fedorahosted.org/389/ticket/47509

Reviewed by: richm(Thanks!!)
(cherry picked from commit 26253234c7b1c69286d8cde8f5e000bae946b6a5)
Ticket 47509 - Cleanallruv jenkins error

Fixed invalid array element that was out of bounds.

https://fedorahosted.org/389/ticket/47509
(cherry picked from commit 24bec17a925c99fe5a655f983e53fc72a34c69ac)
Ticket #47492 - PassSync removes User must change password flag on the Windows side

Bug description: Windows Sync sends password modify even if it is
from PassSync originated on AD.  The modify updates the pwdLastSet
attribute value to non-zero value.  The value 0 indicates the pass-
word must change at next logon on AD.

Fix description: Before sending the password modify, check whether
the current pwdLastSet value is 0 or not.  If it is 0 (means the
password must change), reset pwdLastSet value to 0 along with the
password modify.  This operation replaces the password on AD, but
the password still must change at next logon.

Note: If "password must change at next logon" on the both DS and AD,
the password needs to be changed by the user on the both servers to
enable it on each.

https://fedorahosted.org/389/ticket/47492

Reviewed by Rich (Thank you!!)
(cherry picked from commit f9d1d9e08225e5885b76cede4da677708892ee7a)
Ticket 47513 - tmpfiles.d references /var/lock when they should reference /run/lock

Bug Description:  Due to a possible race condition, the server should be using /run for
                  /var/lock and /var/run

Fix Description:  To be backwards compatible, we first check if /run exists(as it does not exist in rhel 6.x), then set
                  the lock directory to be /run/lock & /var/run to /run, and only write this to our tmpfiles.d configuration.

https://fedorahosted.org/389/ticket/47513

Reviewed by: richm(Thanks!)
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: 389-ds-base-1.3.1
Fix Description: Added a new attribute nsIndexIDListScanLimit to nsIndex
 nsIndexIDListScanLimit: limit=NNN [type=eq[,sub,...]] [flags=ADD[,XXX..]] [values=val[,val...]]
* The limit is the idlistscanlimit to apply.  The value can be -1 (unlimited), 0
(do not use the index at all), or a number to use for the idlistscanlimit.
* The type is a comma-delimited list of index types (eq,sub,etc.) to which the
limit applies.  The index type must be one of the index types configured for the index (you can't configure an id list for type=sub if you do not already have
a substring index configured for the attribute).
* The flags are a comma-delimited list of additional criteria which must match.
** flags ADD - only apply the limit when the index is used in an AND (&) filter
The values are a comma-delimited list of values which must match in the search
filter in order for the limit to be applied.  Since the matches are done one
at a time (we evaluate one filter component at a time), the values will match
if any of the values match.
* The values must be used with only one type at a time.  If values are specified,
type must be specified, and type must be a type that deals with values, such
as eq or sub.  There must be only one type specified - you can't specify values
if you use type=eq,pres or otherwise specify more than one type.  The values
must correspond to the index type (eq, sub), and must correspond to the syntax
of the attribute to which the index is applied - that is, if you have
attribute uidNumber (integer) and it is indexed for eq, you can't specify
type=eq values=abc because "abc" is not integer syntax.
If the values contain spaces, commas, nulls, other values which require
escapes, the LDAP filter escape syntax should be used - backslash '\' followed
by the 2 hex digit code for the character.
The values are processed as if they were filter values - so for "sub" values,
values like "values=*sm*ith*" will be processed as if they were values in a
substring search filter like (sn=*sm*ith*)
* nsIndexIDListScanLimit is multi-valued.  If a search matches more than one
nsIndexIDListScanLimit, the rules are applied in priority order.
The priority is as follows, from highest to lowest:
 * * match type, flags, value
 * * match type, value
 * * match type, flags
 * * match type
 * * match flags
 * For example, if you have
 * dn: cn=objectclass,...
 * objectclass: nsIndex
 * nsIndexType: eq
 * nsIndexIDListScanLimit: limit=0 type=eq flags=AND value=inetOrgPerson
 * nsIndexIDListScanLimit: limit=1 type=eq value=inetOrgPerson
 * nsIndexIDListScanLimit: limit=2 type=eq flags=AND
 * nsIndexIDListScanLimit: limit=3 type=eq
 * nsIndexIDListScanLimit: limit=4 flags=AND
 * nsIndexIDListScanLimit: limit=5
 * If the search filter is (&(objectclass=inetOrgPerson)(uid=foo)) then the limit=0 because all
 *  3 of type, flags, and value match
 * If the search filter is (objectclass=inetOrgPerson) then the limit=1 because type and value match
 *  but flag does not
 * If the search filter is (&(objectclass=posixAccount)(uid=foo)) the the limit=2 because type and
 *  flags match
 * If the search filter is (objectclass=posixAccount) then the limit=3 because only the type matches
 * If the search filter is (&(objectclass=*account*)(objectclass=*)) then the limit=4 because only
 *  flags match but not the types (sub and pres)
 * If the search filter is (objectclass=*account*) then the limit=5 because only the attribute matches
 *  but none of flags, type, or value matches
To add in testing/debugging, the LDAP_DEBUG_BACKLDBM log level is used to
print information about searches which exceed the idlistscanlimit.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: yes - document new attribute
(cherry picked from commit 824b3019beffa5bf2bc5ab2a2a3e579d50833577)
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: master
Fix Description: Fix compiler warning - use bv.bv_val to print syntax
errors in substring filter.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 36f506d05fb940e56c42a6b966f11ad1fdf8cba6)
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: 389-ds-base-1.3.1
Fix Description: This patch broke replication.  The problem is that we cannot
compare the allidslimit to 0 unless the value is explicitly set by
index_get_allids.  The fix is to only return ALLIDS if the allidslimit was
explicitly set to 0 by index_get_allids.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 058d01d7479204a2507dab822cd81e32c37be862)
Ticket 47528 - 389-ds-base built with mozldap can crash from invalid free

Bug Description:  The issue is that the slapi_escape_filter_value() returned string gets
                  freed by the caller. When using mozldap, this function can return the
                  original filter pointer, which can lead to a double free.

Fix Description:  Return a copy of the filter str so it can be safely freed.  Also free
                  the buf if it's not being returned.

https://fedorahosted.org/389/ticket/47528

Reviewed by: nhosoi(Thanks!)
(cherry picked from commit da59cff5e6d27c18bbb02cba98bb99c1bfa37c70)
Ticket #47523 - Set up replcation/agreement before initializing the sub suffix, the sub suffix is not found by ldapsearch

Bug description: If a replication is configured against a backend
before initializing the backend with a suffix entry, an RUV entry
is inserted first with the entryid 1. The RUV entry's entryrdn is
added to the entryrdn index with a suffix entry which is a parent
entry of the RUV entry having a temporary entryid 0, which was to
be replaced with the real entryid when the real suffix entry is
added.  But the replacement code was not executed.

Fix description: When a real suffix is added to the entryrdn index,
it returns DB_KEYEXIST, which used to be ignored by resetting 0
(== SUCCESS).  This patch returns DB_KEYEXIST to the caller and let
_entryrdn_insert_key use the info to replace the temporary entryid
with the real one.  The error code is ignored by the other callers.

https://fedorahosted.org/389/ticket/47523

Reviewed by nkinder (Thanks!).
(cherry picked from commit e6eab21920a0374eb356da3d1f041312c6857ecd)
Ticket 47510 - 389-ds-base does not compile against MozLDAP libraries

Used #ifdef's to remove openldap specific API calls.

Thanks to mvocu for providing the initial patch.

https://fedorahosted.org/389/ticket/47510

Reviewed by: rmeggins(Thanks!)
(cherry picked from commit 069657f1b3c04390c438066bd9ddc8c6a79f1dd0)
Ticket #47534 - RUV tombstone search with scope "one" doesn`t work

Bug description: slapi_sdn_scope_test_ext examines the given dn
satisfies the scope against the base dn.  Onelevel case for the
tombstone DN ignores the nsuniqueid leaf rdn before checking the
scope which should not be done for the RUV tombstone.

Fix description: This patch correctly handles the RUV tombstone
as an exception.

https://fedorahosted.org/389/ticket/47534

Reviewed by rmeggins (Thanks, Rich!)
(cherry picked from commit fbece32d94515bc1ca3889c3d7c0691bbeec3ab0)
bump version to 1.3.1.10
Ticket 47510 - Repl Sync does not compile against MozLDAP libraries

Repl Sync uses openldap extensively, so we had to bring over many defines.  Also
fixed some other warnings.

https://fedorahosted.org/389/ticket/47510

Reviewed by: rmeggins(Thanks!)
Ticket 47510 - remove unnecessary typedef

https://fedorahosted.org/389/ticket/47510
(cherry picked from commit d876e6dedcc43e7c7acd775da0749b636649b00a)
Ticket 47513 - Refine the check for @localrundir@

Bug Description:  if @localrundir@ is not set, then the wrong value is used for tmpfiles.d

Fix Description:  Improve the checking/setting of the localrundir.  set_path_attribute() is not
                  the correct subroutine to call as it can incorrectly add the prefix again
                  if @localrundir@ is not set.  Created new function to just add the prefix -
                  only if @localrundir@ is set.

https://fedorahosted.org/389/ticket/47513

Reviewed by: richm(Thanks!)
(cherry picked from commit 9e38a3be20c87141221b52ce992a713cba9eae3b)
Ticket 47513 - Set localrundir outside of the "with-fhs" block

Description:  Previously if you built 389 without --with-fhs, localrundir would
              not be set.  Set localrundir above the with-fhs section, and then
              only reset as needed.

https://fedorahosted.org/389/ticket/47531

Reviewed by: rmeggins(Thanks!)
bump version to 1.3.1.11
Ticket 47354 - Indexed search are logged with 'notes=U' in the access logs

Bug Description:
	In the access log, the flag "notes=U" is present to warn the customer that the provided filter contains not
	indexed components. This flag is set independently of the fact that all entries of the database will be
	evaluated against the filter. So it can not be used to determine if all the entries have been scanned.
	Evaluated all entries is usually a performance hit for large database containing millions of entries.

Fix Description:
	The fix consist to introduce a new flag 'notes=A' at the top level (ldbm_back_search) only when the candidate
	list is ALLID and all entries will be scanned.
	The fix also contains the update of logconv.pl to take into account this change.

https://fedorahosted.org/389/ticket/47354

Reviewed by: Rich Megginson and Mark Reynolds (thanks to you !)

Platforms tested: Fedora 17

Flag Day: no

Doc impact: no
(cherry picked from commit f2b5a975c2268b6e99bf27b45a7380d295807d81)
Ticket #47387 - improve logconv.pl performance with large access logs

https://fedorahosted.org/389/ticket/47387
Reviewed by: mreynolds (Thanks!)
Branch: master
Fix Description: The primary fix is to use tied hashes and arrays backed by
DB_File database files.  Hashes use DB_HASH and arrays use DB_RECNO.  These
fixes cut down the time by a factor of 10 or more, and considerably reduce
the size of the disk files.  There is still room for optimization as given
by the Devel::NYTProf package, but this should at least give us acceptable
performance.
I also did some perl "linting" - running with use warnings and use strict -
which revealed a lot of code which was cleaned up.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
Ticket #47387 - improve logconv.pl performance with large access logs

https://fedorahosted.org/389/ticket/47387
Reviewed by: mreynolds (Thanks!)
Branch: master
Fix Description: Fix some additional uninitialized variable warnings
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 8a23f5e0af41291d28c4ce7251d9bb2e94a5994c)