Ticket #47492 - PassSync removes User must change password flag on the Windows side

Bug description: Windows Sync sends password modify even if it is
from PassSync originated on AD.  The modify updates the pwdLastSet
attribute value to non-zero value.  The value 0 indicates the pass-
word must change at next logon on AD.

Fix description: Before sending the password modify, check whether
the current pwdLastSet value is 0 or not.  If it is 0 (means the
password must change), reset pwdLastSet value to 0 along with the
password modify.  This operation replaces the password on AD, but
the password still must change at next logon.

Note: If "password must change at next logon" on the both DS and AD,
the password needs to be changed by the user on the both servers to
enable it on each.

https://fedorahosted.org/389/ticket/47492

Reviewed by Rich (Thank you!!)
(cherry picked from commit f9d1d9e08225e5885b76cede4da677708892ee7a)
(cherry picked from commit 8d34f77f6d8d3c83dce1f29e6df709df1adef09d)
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: 389-ds-base-1.2.11
Fix Description: Added a new attribute nsIndexIDListScanLimit to nsIndex
 nsIndexIDListScanLimit: limit=NNN [type=eq[,sub,...]] [flags=ADD[,XXX..]] [values=val[,val...]]
* The limit is the idlistscanlimit to apply.  The value can be -1 (unlimited), 0
(do not use the index at all), or a number to use for the idlistscanlimit.
* The type is a comma-delimited list of index types (eq,sub,etc.) to which the
limit applies.  The index type must be one of the index types configured for the index (you can't configure an id list for type=sub if you do not already have
a substring index configured for the attribute).
* The flags are a comma-delimited list of additional criteria which must match.
** flags ADD - only apply the limit when the index is used in an AND (&) filter
The values are a comma-delimited list of values which must match in the search
filter in order for the limit to be applied.  Since the matches are done one
at a time (we evaluate one filter component at a time), the values will match
if any of the values match.
* The values must be used with only one type at a time.  If values are specified,
type must be specified, and type must be a type that deals with values, such
as eq or sub.  There must be only one type specified - you can't specify values
if you use type=eq,pres or otherwise specify more than one type.  The values
must correspond to the index type (eq, sub), and must correspond to the syntax
of the attribute to which the index is applied - that is, if you have
attribute uidNumber (integer) and it is indexed for eq, you can't specify
type=eq values=abc because "abc" is not integer syntax.
If the values contain spaces, commas, nulls, other values which require
escapes, the LDAP filter escape syntax should be used - backslash '\' followed
by the 2 hex digit code for the character.
The values are processed as if they were filter values - so for "sub" values,
values like "values=*sm*ith*" will be processed as if they were values in a
substring search filter like (sn=*sm*ith*)
* nsIndexIDListScanLimit is multi-valued.  If a search matches more than one
nsIndexIDListScanLimit, the rules are applied in priority order.
The priority is as follows, from highest to lowest:
 * * match type, flags, value
 * * match type, value
 * * match type, flags
 * * match type
 * * match flags
 * For example, if you have
 * dn: cn=objectclass,...
 * objectclass: nsIndex
 * nsIndexType: eq
 * nsIndexIDListScanLimit: limit=0 type=eq flags=AND value=inetOrgPerson
 * nsIndexIDListScanLimit: limit=1 type=eq value=inetOrgPerson
 * nsIndexIDListScanLimit: limit=2 type=eq flags=AND
 * nsIndexIDListScanLimit: limit=3 type=eq
 * nsIndexIDListScanLimit: limit=4 flags=AND
 * nsIndexIDListScanLimit: limit=5
 * If the search filter is (&(objectclass=inetOrgPerson)(uid=foo)) then the limit=0 because all
 *  3 of type, flags, and value match
 * If the search filter is (objectclass=inetOrgPerson) then the limit=1 because type and value match
 *  but flag does not
 * If the search filter is (&(objectclass=posixAccount)(uid=foo)) the the limit=2 because type and
 *  flags match
 * If the search filter is (objectclass=posixAccount) then the limit=3 because only the type matches
 * If the search filter is (&(objectclass=*account*)(objectclass=*)) then the limit=4 because only
 *  flags match but not the types (sub and pres)
 * If the search filter is (objectclass=*account*) then the limit=5 because only the attribute matches
 *  but none of flags, type, or value matches
To add in testing/debugging, the LDAP_DEBUG_BACKLDBM log level is used to
print information about searches which exceed the idlistscanlimit.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: yes - document new attribute
(cherry picked from commit 824b3019beffa5bf2bc5ab2a2a3e579d50833577)
(cherry picked from commit b348886030318cd43855ae439ec3f30f898b8cd4)
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: master
Fix Description: Fix compiler warning - use bv.bv_val to print syntax
errors in substring filter.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 36f506d05fb940e56c42a6b966f11ad1fdf8cba6)
(cherry picked from commit e95d7d6bd893d777c88be1bd847996f299621d99)
(cherry picked from commit 527c3e49ed14bc585f29ed873e71c6f551ad193b)
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: 389-ds-base-1.2.11
Fix Description: This patch broke replication.  The problem is that we cannot
compare the allidslimit to 0 unless the value is explicitly set by
index_get_allids.  The fix is to only return ALLIDS if the allidslimit was
explicitly set to 0 by index_get_allids.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 058d01d7479204a2507dab822cd81e32c37be862)
(cherry picked from commit e5405e627439ccaf370d90c42e65c0a987e33e73)
(cherry picked from commit c96eaa03738e36b7cde0656edfbe6c4769847b52)
Ticket #47523 - Set up replcation/agreement before initializing the sub suffix, the sub suffix is not found by ldapsearch

Bug description: If a replication is configured against a backend
before initializing the backend with a suffix entry, an RUV entry
is inserted first with the entryid 1. The RUV entry's entryrdn is
added to the entryrdn index with a suffix entry which is a parent
entry of the RUV entry having a temporary entryid 0, which was to
be replaced with the real entryid when the real suffix entry is
added.  But the replacement code was not executed.

Fix description: When a real suffix is added to the entryrdn index,
it returns DB_KEYEXIST, which used to be ignored by resetting 0
(== SUCCESS).  This patch returns DB_KEYEXIST to the caller and let
_entryrdn_insert_key use the info to replace the temporary entryid
with the real one.  The error code is ignored by the other callers.

https://fedorahosted.org/389/ticket/47523

Reviewed by nkinder (Thanks!).
(cherry picked from commit e6eab21920a0374eb356da3d1f041312c6857ecd)
(cherry picked from commit 6b35dc7b04da023cc14045cc6dd1f5e304cf265f)
Ticket 47489 - Under specific values of nsDS5ReplicaName, replication may get broken or updates missing

Bug Description:
	If the 'nsDS5ReplicaName' (of a replica), contains the database suffix (e.g. 'db', 'db3' or 'db4).
	Then replication plugin fails to open the changelog. It could conduct to changelog being recreated or some
	last updates to be corrupted.
	A consequence that I can reproduce, is that some updates may be removed from the changelog and missing
	updates on consumers.
	This could conduct to replication break, if for example an entry created is not replicated and then later updated.

Fix Description:
	The fix consist to use 'PL_strrstr' rather than 'strstr' to check the database suffix is valid

https://fedorahosted.org/389/ticket/47489

Reviewed by: Rich Megginson (thanks Rich !)

Platforms tested: Fedora 17

Flag Day: no

Doc impact: no
(cherry picked from commit 7a7609d88caf9c0971e694d7eeb78f30aea7fec9)
(cherry picked from commit ac8aad8260d3e5ed403e2d4a9967447a97925ba7)
Ticket #47534 - RUV tombstone search with scope "one" doesn`t work

Bug description: slapi_sdn_scope_test_ext examines the given dn
satisfies the scope against the base dn.  Onelevel case for the
tombstone DN ignores the nsuniqueid leaf rdn before checking the
scope which should not be done for the RUV tombstone.

Fix description: This patch correctly handles the RUV tombstone
as an exception.

https://fedorahosted.org/389/ticket/47534

Reviewed by rmeggins (Thanks, Rich!)
Ticket #356 - RFE - Track bind info

Bug Description:  Have a way to gather bind stats (operations and IP addresses).

Fix Description:  Added a new option "-B,--bind" which takes one of three arguemnts:
                   [1]  ALL - report on all Bind DN's
                   [2]  ANONYMOUS - report just on anonymous binds
                   [3]  "Bind DN" - report on the specified bind dn (uid=mark,dc=example,dc=com)

https://fedorahosted.org/389/ticket/356

reviewed by:
(cherry picked from commit 9918105f615084f65a47fad693e02fce53bb7d72)
Ticket 539 - logconv.pl should handle microsecond timing

Bug Description:  logconv.pl can not handle microsecond etimes

Fix Description:  Update the regular expressions to handle the decimal point "."

                  Also did a little code cleanup.

https://fedorahosted.org/389/ticket/539

Reviewed by: nhosoi(Thanks!)
(cherry picked from commit 8965dda8d4023ab36f0a404a874c657d78631e9d)
(cherry picked from commit 67c5f1cd1584cd352da7b8db2520c268431d7812)
Ticket 471 - logconv.pl tool removes the access logs contents if "-M" is not correctly used

Bug Description:  If you fail to specifiy the output file, the access log is overwritten.

Fix Description:  Check if the proper files have been provbided to the script, and report error otherwise.

https://fedorahosted.org/389/ticket/471

Reviewed by: nhosoi(Thanks!)
(cherry picked from commit 89fb3420c9137cb2c9813fec527b89fec3b6d63e)
(cherry picked from commit 9f0c9392ac7e27193831098f1efd4b7047432cd7)
TIcket 419 - logconv.pl - improve memory management

Remove old logconv.pl script - to make room for the new rewrite of logconv.pl

https://fedorahosted.org/389/ticket/419
(cherry picked from commit 472373506eeca676dbbf9b9f56c2248761068752)
(cherry picked from commit fc3c9192d37631948ca17fec3cd9bd2fed1ffdbc)
Ticket 419 - logconv.pl - improve memory management

Bug Description:  logconv.pl memory usage is very high, and doesn't
                  appear to be efficient.

Fix Description:  For verbose stats/hashes/arrays we now write this
                  information to disk, and read it in and process it
                  when printing the report.  This dramatically improved
                  memory growth when dealing with larger and larger log sets.

                  Also updated the oids which some were incorrect, but
                  many new oids were missing.  Added stats for "Paged Searches",
                  and "Max BER size exceeded".

                  Finally, renamed most variables to be readable names,
                  removed redundant counters, and other generic imrpovements.

https://fedorahosted.org/389/ticket/419

Reviewed by: Noriko(Thanks!)
(cherry picked from commit 16651d642bf6f7ba624f62cac867b495253b4f3f)
(cherry picked from commit 9866682da7fb35e16922d94404c4265701fc797a)
Ticket 611 - logconv.pl missing stats for StartTLS, LDAPI, and AUTOBIND

RFE Description:  Previously there were no stats for StartTLS, LDAPI,
                  and AUTOTBIND connections/binds.

https://fedorahosted.org/389/ticket/611

Reviewed by: richm(Thanks!)
(cherry picked from commit 59ae73c25c316c81c1c18ea702c8e793c922e1d8)
(cherry picked from commit 9f382458ee799e08f9ceddab73111011ea15e3a4)
Ticket #47336 - logconv.pl -m not working for all stats

https://fedorahosted.org/389/ticket/47336
Reviewed by: nhosoi (Thanks!)
Branch: 389-ds-base-1.3.1
Fix Description: Code was using $reportStat instead of $reportStats.  Also,
some fields such as connections, ssl connections were not being kept.
Finally, some headers were missing.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 27f2595bfd54c85bb169ac45cac2cdbfd4de17fc)
(cherry picked from commit 26bd66a10ab942c8782294d89a9e3e1768bc7936)
(cherry picked from commit 54193cb97b9157d0da785719a3d91f0f58cfb5e7)
Ticket #47341 - logconv.pl -m time calculation is wrong

https://fedorahosted.org/389/ticket/47341
Reviewed by: nkinder (Thanks!)
Branch: 389-ds-base-1.3.1
Fix Description: The variable name is $hr, not $hours.  The regex for the
sign in the timezone should have been (.) not (?).
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 87f5ef5a97437cc92afe2496cd06e5d2ec71d9ff)
(cherry picked from commit 1878bbfef7ae76ed18d709d5414e21cbd0961a38)
(cherry picked from commit eea06437c2be14fa9ab70dc983e2ab138f8d62c8)
Ticket #47348 - add etimes to per second/minute stats

https://fedorahosted.org/389/ticket/47348
Reviewed by: mreynolds (Thanks!)
Branch: 389-ds-base-1.3.1
Fix Description: Add an "ElapsedTime" column to the -m/-M output.  This column
is the cumulative etimes of all operations during that time period.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: yes
(cherry picked from commit 9813b219ad2aa7b7194d9295dd529e12889a8534)

Ticket #47349 - DS instance crashes under a high load

https://fedorahosted.org/389/ticket/47349
Reviewed by: nkinder (Thanks!)
Branch: 389-ds-base-1.3.1
Fix Description: handle_new_connection initializes the connection object,
then calls connection_table_move_connection_on_to_active_list to put it
on the list of active connections, then unlocks the c_mutex, then calls
connection_new_private to allocate c_private.  If another thread
interrupts after the conn has been moved to the active list, but before
c_private has been allocated, the new conn will be available via
connection_table_iterate_active_connections where table_iterate_function
will attempt to dereference the NULL c_private.
The fix is to move connection_new_private inside the c_mutex lock, and to
move connection_table_move_connection_on_to_active_list to be the very last
thing before releasing the c_mutex lock.  Once the conn is on the active
list it is live and we cannot do anything else to it.
Note: I have still not been able to reproduce the problem in a non-debug
optimized build.
Platforms tested: RHEL6 x86_64
Note: Before patch, server would crash within 5 minutes.  After patch, server
has been running for several days in customer environment.
Flag Day: no
Doc impact: no
(cherry picked from commit 05d209432571dc64b242ae47113ae4cbb43607d2)
(cherry picked from commit c7fc203ffe09809e430ff1217ec352db95d636a0)
(cherry picked from commit 02a70f7e8f3bf5deac9039b51c31a206e628e9e1)
Ticket 47447 - logconv.pl man page missing -m,-M,-B,-D

Fix Description:  Added the missing info, and also updated the main arguments
                  with their long option names.

https://fedorahosted.org/389/ticket/47447

Reviewed by: nhosoi(Thanks!)
(cherry picked from commit 4ea4c8bf91f5c28bd643b0c06f590ec164156358)
(cherry picked from commit ed245c45813255400c6962707f7269caa253e953)
(cherry picked from commit 65eb63b8a35afe5026386b1dda6eb321f52e18f2)
Ticket 47461 - logconv.pl - Use of comma-less variable list is deprecated

Description:  In newer versions of perl commas are needed to separate variables
              when "formatting" output.  Also fixed a typo.

https://fedorahosted.org/389/ticket/47461

Reviewed by: nhosoi(Thanks!)
(cherry picked from commit 7e9cae5fa2292168e88e87eee2f9249171a1e9d8)
(cherry picked from commit 3e7ee7ca886feb56b3b26df97882c40d81bfff55)
(cherry picked from commit ba61cebeb12b4d11706a80f746f2fd97f5b5f7f9)
Ticket 47354 - Indexed search are logged with 'notes=U' in the access logs

Bug Description:
	In the access log, the flag "notes=U" is present to warn the customer that the provided filter contains not
	indexed components. This flag is set independently of the fact that all entries of the database will be
	evaluated against the filter. So it can not be used to determine if all the entries have been scanned.
	Evaluated all entries is usually a performance hit for large database containing millions of entries.

Fix Description:
	The fix consist to introduce a new flag 'notes=A' at the top level (ldbm_back_search) only when the candidate
	list is ALLID and all entries will be scanned.
	The fix also contains the update of logconv.pl to take into account this change.

https://fedorahosted.org/389/ticket/47354

Reviewed by: Rich Megginson and Mark Reynolds (thanks to you !)

Platforms tested: Fedora 17

Flag Day: no

Doc impact: no
(cherry picked from commit f2b5a975c2268b6e99bf27b45a7380d295807d81)
(cherry picked from commit 15c77ad6e757625cd8a57bb5cde5e8d697bf5b25)
(cherry picked from commit fb1c15d8b89c632e0e45c885334d47b6bbb17730)
Ticket #47387 - improve logconv.pl performance with large access logs

https://fedorahosted.org/389/ticket/47387
Reviewed by: mreynolds (Thanks!)
Branch: master
Fix Description: The primary fix is to use tied hashes and arrays backed by
DB_File database files.  Hashes use DB_HASH and arrays use DB_RECNO.  These
fixes cut down the time by a factor of 10 or more, and considerably reduce
the size of the disk files.  There is still room for optimization as given
by the Devel::NYTProf package, but this should at least give us acceptable
performance.
I also did some perl "linting" - running with use warnings and use strict -
which revealed a lot of code which was cleaned up.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 8e35cc8b418bc46fbeb28f4378d24f9c2cab0353)
(cherry picked from commit 481f2e3aec2834f342c01919d91eb6cfd80f6ca1)