Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: 389-ds-base-1.3.0
Fix Description: Added a new attribute nsIndexIDListScanLimit to nsIndex
 nsIndexIDListScanLimit: limit=NNN [type=eq[,sub,...]] [flags=ADD[,XXX..]] [values=val[,val...]]
* The limit is the idlistscanlimit to apply.  The value can be -1 (unlimited), 0
(do not use the index at all), or a number to use for the idlistscanlimit.
* The type is a comma-delimited list of index types (eq,sub,etc.) to which the
limit applies.  The index type must be one of the index types configured for the index (you can't configure an id list for type=sub if you do not already have
a substring index configured for the attribute).
* The flags are a comma-delimited list of additional criteria which must match.
** flags ADD - only apply the limit when the index is used in an AND (&) filter
The values are a comma-delimited list of values which must match in the search
filter in order for the limit to be applied.  Since the matches are done one
at a time (we evaluate one filter component at a time), the values will match
if any of the values match.
* The values must be used with only one type at a time.  If values are specified,
type must be specified, and type must be a type that deals with values, such
as eq or sub.  There must be only one type specified - you can't specify values
if you use type=eq,pres or otherwise specify more than one type.  The values
must correspond to the index type (eq, sub), and must correspond to the syntax
of the attribute to which the index is applied - that is, if you have
attribute uidNumber (integer) and it is indexed for eq, you can't specify
type=eq values=abc because "abc" is not integer syntax.
If the values contain spaces, commas, nulls, other values which require
escapes, the LDAP filter escape syntax should be used - backslash '\' followed
by the 2 hex digit code for the character.
The values are processed as if they were filter values - so for "sub" values,
values like "values=*sm*ith*" will be processed as if they were values in a
substring search filter like (sn=*sm*ith*)
* nsIndexIDListScanLimit is multi-valued.  If a search matches more than one
nsIndexIDListScanLimit, the rules are applied in priority order.
The priority is as follows, from highest to lowest:
 * * match type, flags, value
 * * match type, value
 * * match type, flags
 * * match type
 * * match flags
 * For example, if you have
 * dn: cn=objectclass,...
 * objectclass: nsIndex
 * nsIndexType: eq
 * nsIndexIDListScanLimit: limit=0 type=eq flags=AND value=inetOrgPerson
 * nsIndexIDListScanLimit: limit=1 type=eq value=inetOrgPerson
 * nsIndexIDListScanLimit: limit=2 type=eq flags=AND
 * nsIndexIDListScanLimit: limit=3 type=eq
 * nsIndexIDListScanLimit: limit=4 flags=AND
 * nsIndexIDListScanLimit: limit=5
 * If the search filter is (&(objectclass=inetOrgPerson)(uid=foo)) then the limit=0 because all
 *  3 of type, flags, and value match
 * If the search filter is (objectclass=inetOrgPerson) then the limit=1 because type and value match
 *  but flag does not
 * If the search filter is (&(objectclass=posixAccount)(uid=foo)) the the limit=2 because type and
 *  flags match
 * If the search filter is (objectclass=posixAccount) then the limit=3 because only the type matches
 * If the search filter is (&(objectclass=*account*)(objectclass=*)) then the limit=4 because only
 *  flags match but not the types (sub and pres)
 * If the search filter is (objectclass=*account*) then the limit=5 because only the attribute matches
 *  but none of flags, type, or value matches
To add in testing/debugging, the LDAP_DEBUG_BACKLDBM log level is used to
print information about searches which exceed the idlistscanlimit.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: yes - document new attribute
(cherry picked from commit 824b3019beffa5bf2bc5ab2a2a3e579d50833577)
(cherry picked from commit b348886030318cd43855ae439ec3f30f898b8cd4)
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: master
Fix Description: Fix compiler warning - use bv.bv_val to print syntax
errors in substring filter.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 36f506d05fb940e56c42a6b966f11ad1fdf8cba6)
(cherry picked from commit e95d7d6bd893d777c88be1bd847996f299621d99)
Ticket #47504 idlistscanlimit per index/type/value

https://fedorahosted.org/389/ticket/47504
Reviewed by: nhosoi (Thanks!)
Branch: 389-ds-base-1.3.0
Fix Description: This patch broke replication.  The problem is that we cannot
compare the allidslimit to 0 unless the value is explicitly set by
index_get_allids.  The fix is to only return ALLIDS if the allidslimit was
explicitly set to 0 by index_get_allids.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 058d01d7479204a2507dab822cd81e32c37be862)
(cherry picked from commit e5405e627439ccaf370d90c42e65c0a987e33e73)
Ticket 539 - logconv.pl should handle microsecond timing

Bug Description:  logconv.pl can not handle microsecond etimes

Fix Description:  Update the regular expressions to handle the decimal point "."

                  Also did a little code cleanup.

https://fedorahosted.org/389/ticket/539

Reviewed by: nhosoi(Thanks!)
(cherry picked from commit 8965dda8d4023ab36f0a404a874c657d78631e9d)
Ticket 471 - logconv.pl tool removes the access logs contents if "-M" is not correctly used

Bug Description:  If you fail to specifiy the output file, the access log is overwritten.

Fix Description:  Check if the proper files have been provbided to the script, and report error otherwise.

https://fedorahosted.org/389/ticket/471

Reviewed by: nhosoi(Thanks!)
(cherry picked from commit 89fb3420c9137cb2c9813fec527b89fec3b6d63e)
TIcket 419 - logconv.pl - improve memory management

Remove old logconv.pl script - to make room for the new rewrite of logconv.pl

https://fedorahosted.org/389/ticket/419
(cherry picked from commit 472373506eeca676dbbf9b9f56c2248761068752)
Ticket 419 - logconv.pl - improve memory management

Bug Description:  logconv.pl memory usage is very high, and doesn't
                  appear to be efficient.

Fix Description:  For verbose stats/hashes/arrays we now write this
                  information to disk, and read it in and process it
                  when printing the report.  This dramatically improved
                  memory growth when dealing with larger and larger log sets.

                  Also updated the oids which some were incorrect, but
                  many new oids were missing.  Added stats for "Paged Searches",
                  and "Max BER size exceeded".

                  Finally, renamed most variables to be readable names,
                  removed redundant counters, and other generic imrpovements.

https://fedorahosted.org/389/ticket/419

Reviewed by: Noriko(Thanks!)
(cherry picked from commit 16651d642bf6f7ba624f62cac867b495253b4f3f)
Ticket 611 - logconv.pl missing stats for StartTLS, LDAPI, and AUTOBIND

RFE Description:  Previously there were no stats for StartTLS, LDAPI,
                  and AUTOTBIND connections/binds.

https://fedorahosted.org/389/ticket/611

Reviewed by: richm(Thanks!)
(cherry picked from commit 59ae73c25c316c81c1c18ea702c8e793c922e1d8)
Ticket #47336 - logconv.pl -m not working for all stats

https://fedorahosted.org/389/ticket/47336
Reviewed by: nhosoi (Thanks!)
Branch: 389-ds-base-1.3.1
Fix Description: Code was using $reportStat instead of $reportStats.  Also,
some fields such as connections, ssl connections were not being kept.
Finally, some headers were missing.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 27f2595bfd54c85bb169ac45cac2cdbfd4de17fc)
(cherry picked from commit 26bd66a10ab942c8782294d89a9e3e1768bc7936)
Ticket #47341 - logconv.pl -m time calculation is wrong

https://fedorahosted.org/389/ticket/47341
Reviewed by: nkinder (Thanks!)
Branch: 389-ds-base-1.3.1
Fix Description: The variable name is $hr, not $hours.  The regex for the
sign in the timezone should have been (.) not (?).
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 87f5ef5a97437cc92afe2496cd06e5d2ec71d9ff)
(cherry picked from commit 1878bbfef7ae76ed18d709d5414e21cbd0961a38)
Ticket #47348 - add etimes to per second/minute stats

https://fedorahosted.org/389/ticket/47348
Reviewed by: mreynolds (Thanks!)
Branch: 389-ds-base-1.3.1
Fix Description: Add an "ElapsedTime" column to the -m/-M output.  This column
is the cumulative etimes of all operations during that time period.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: yes
(cherry picked from commit 9813b219ad2aa7b7194d9295dd529e12889a8534)

Ticket #47349 - DS instance crashes under a high load

https://fedorahosted.org/389/ticket/47349
Reviewed by: nkinder (Thanks!)
Branch: 389-ds-base-1.3.1
Fix Description: handle_new_connection initializes the connection object,
then calls connection_table_move_connection_on_to_active_list to put it
on the list of active connections, then unlocks the c_mutex, then calls
connection_new_private to allocate c_private.  If another thread
interrupts after the conn has been moved to the active list, but before
c_private has been allocated, the new conn will be available via
connection_table_iterate_active_connections where table_iterate_function
will attempt to dereference the NULL c_private.
The fix is to move connection_new_private inside the c_mutex lock, and to
move connection_table_move_connection_on_to_active_list to be the very last
thing before releasing the c_mutex lock.  Once the conn is on the active
list it is live and we cannot do anything else to it.
Note: I have still not been able to reproduce the problem in a non-debug
optimized build.
Platforms tested: RHEL6 x86_64
Note: Before patch, server would crash within 5 minutes.  After patch, server
has been running for several days in customer environment.
Flag Day: no
Doc impact: no
(cherry picked from commit 05d209432571dc64b242ae47113ae4cbb43607d2)
(cherry picked from commit c7fc203ffe09809e430ff1217ec352db95d636a0)
Ticket 47447 - logconv.pl man page missing -m,-M,-B,-D

Fix Description:  Added the missing info, and also updated the main arguments
                  with their long option names.

https://fedorahosted.org/389/ticket/47447

Reviewed by: nhosoi(Thanks!)
(cherry picked from commit 4ea4c8bf91f5c28bd643b0c06f590ec164156358)
(cherry picked from commit ed245c45813255400c6962707f7269caa253e953)
Ticket 47461 - logconv.pl - Use of comma-less variable list is deprecated

Description:  In newer versions of perl commas are needed to separate variables
              when "formatting" output.  Also fixed a typo.

https://fedorahosted.org/389/ticket/47461

Reviewed by: nhosoi(Thanks!)
(cherry picked from commit 7e9cae5fa2292168e88e87eee2f9249171a1e9d8)
(cherry picked from commit 3e7ee7ca886feb56b3b26df97882c40d81bfff55)
Ticket 47354 - Indexed search are logged with 'notes=U' in the access logs

Bug Description:
	In the access log, the flag "notes=U" is present to warn the customer that the provided filter contains not
	indexed components. This flag is set independently of the fact that all entries of the database will be
	evaluated against the filter. So it can not be used to determine if all the entries have been scanned.
	Evaluated all entries is usually a performance hit for large database containing millions of entries.

Fix Description:
	The fix consist to introduce a new flag 'notes=A' at the top level (ldbm_back_search) only when the candidate
	list is ALLID and all entries will be scanned.
	The fix also contains the update of logconv.pl to take into account this change.

https://fedorahosted.org/389/ticket/47354

Reviewed by: Rich Megginson and Mark Reynolds (thanks to you !)

Platforms tested: Fedora 17

Flag Day: no

Doc impact: no
(cherry picked from commit f2b5a975c2268b6e99bf27b45a7380d295807d81)
(cherry picked from commit 15c77ad6e757625cd8a57bb5cde5e8d697bf5b25)
Ticket #47387 - improve logconv.pl performance with large access logs

https://fedorahosted.org/389/ticket/47387
Reviewed by: mreynolds (Thanks!)
Branch: master
Fix Description: The primary fix is to use tied hashes and arrays backed by
DB_File database files.  Hashes use DB_HASH and arrays use DB_RECNO.  These
fixes cut down the time by a factor of 10 or more, and considerably reduce
the size of the disk files.  There is still room for optimization as given
by the Devel::NYTProf package, but this should at least give us acceptable
performance.
I also did some perl "linting" - running with use warnings and use strict -
which revealed a lot of code which was cleaned up.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 8e35cc8b418bc46fbeb28f4378d24f9c2cab0353)
Ticket #47387 - improve logconv.pl performance with large access logs

https://fedorahosted.org/389/ticket/47387
Reviewed by: mreynolds (Thanks!)
Branch: master
Fix Description: Fix some additional uninitialized variable warnings
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 8a23f5e0af41291d28c4ce7251d9bb2e94a5994c)
(cherry picked from commit 7a107bdb67dad58728977b3f410df1d94c6cdb91)
Ticket 47520 - Fix various issues with logconv.pl

Bug Description:  When testing an access log that contains "internal" connections/ops, many
                  warning messages are created.  Other warnings are generated because the
                  DB_File arrays need to have thier elements checked before comparing to other
                  strings.

Fix Description:  Improve value checking.  Add support for connection numbers that are words (Internal),
                  and negative values for op ids(-1).

https://fedorahosted.org/389/ticket/47520

Reviewed by: richm(Thanks!)
(cherry picked from commit 8b153e999ac6dc4869498a78da103a07db49d28e)
(cherry picked from commit 2c5d52b7a4aef8317cc9e34a9b1153c74620010d)
Ticket #47501 logconv.pl uses /var/tmp for BDB temp files

https://fedorahosted.org/389/ticket/47501
Reviewed by: mreynolds (Thanks!)
Branch: master
Fix Description: Tied arrays with DB_RECNO require a backing text file.  By
default this goes in /var/tmp.  This can be controlled with RECNOINFO bfname.
However, the use of arrays was killing performance.  This patch replaces
all arrays with tied hashes.  In addition, this patch fixes the notes=U
handling to also handle notes=U,P and notes=A,P
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 4d20922423c9e22c858e0be7dd317591631cf357)
(cherry picked from commit d890e65b4ab15d0a836dbb248da8606867938630)
Ticket 47533 logconv: some stats do not work across server restarts

https://fedorahosted.org/389/ticket/47533
Reviewed by: mreynolds (Thanks!)
Branch: master
Fix Description: Most of the stat hashes use connid,opid as the hash key.
The connid,opid are not unique because the server recycles connection ids
when the server restarts.  The solution is to use a triplet of
$serverRestartCount,$connid,$opid
as the hash key.  This uniquely identifies a particular log event over
restarts.  There is also some additional cleanup that needs to be done when
the server is reset - returning fds, marking connections as closed, etc.
Also fixed some gmtime handling - we were doing too many conversions.
This fix adds another command line option - -U - for a summary of search
filters which are unindexed or have unindexed components.  The full detailed
report is just too much data.
Platforms tested: RHEL6 x86_64
Flag Day: no
Doc impact: no
(cherry picked from commit 92ac8f48c1e5a8a049f4cfd1912fd570c628408f)
(cherry picked from commit e38685be139c2420cc969499324405a30203122a)
Coverity fixes - 12023, 12024, and 12025

. 12023 - Ignoring number of bytes read
  basicInit (ldclt.c):
  The return value from fread was ignored and not used for copying
  the read content from buffer to mctx.attrplFileContent.
. 12024 - Resource leak
  roles_cache_create_object_from_entry (roles_cache.c):
  When an error occurred, filter_attr_value was not freed.

Reviewed by rmeggins (Thank you, Rich!!)
(cherry picked from commit f702868012ac1f9deb1cb92d51cdfd793353e836)
(cherry picked from commit 99f7b65e4bd35ce5d2c24a05178cfca4a44645db)