idle: no limits for logged in, clear between connections
annotate: store specialuse annotation in the DB
mboxlist: readwrite new format
mailbox: cleanup correctly when DELETED records retained
mboxlist: actually be able to remove RESERVED entries
mboxlist: legacy specialuse upgrade from 2.5-pre format
mboxevent: don't CANCEL all events because of specialuse stuff
mbexamine: print CID (will be zero without Conversations code backport)
Remove sieve action string

We use sieve to forward messages to a notification
daemon which can send SMSes, and we don't want them
filled with information about the actions which
Sieve took.

This patch stops the action_string being appended to
the message sent to the notify daemon.
GUID IMAP COMMANDS

This patch factors out stuff we used to have in the old MD5UUIDs
patch, the following FETCH responses in imapd:

FETCH DIGEST.SHA1 => 40 character hex string (message sha1)
FETCH RFC822.SHA1 => 40 character hex string (message sha1, calculated)
FETCH RFC822.FILESIZE => size of actual file on disk (via stat or mmap)

It also adds a capability string item: "DIGEST=SHA1"

Totally non-standard of course, but way useful for our replication checking
scripts.  Embrace and extend 'r' us.

Anyone feel like writing an RFC for fetching the digest of a message via
IMAP?  If the server calculated it on delivery and cached it then you'd have
a great way to clean up after a UIDVALIDITY change or other destabilising event
without having to fetch every message again.
Tool to build debian package directory
cyrusdb: export cyrusdb_create
hammer_cyrusdb: now with more generic
Fatal -> abort in common daemons
Fastmail Secrets (no rated)
mboxname: add nextmodseq and nextuidvalidity APIs

Merge summary:

    merged in "mboxname: initialise fd"
    merged in "use per-user meta files to serialize uidvalidity and highestmodseq"
    merged in "only use per-user modseq and uidvalidity if conversations are enabled"
    merged in "managed values: add set API and use in replication"
    merged in "mboxname_setval: make sync_server value setting work correctly"
    merged in "sync: handle highestmodseq correctly on mailbox create"
    merged in "cunit: test mboxname_nextmodseq (currently broken)"
    merged in "Fix the mboxname.nextmodseq test."
    merged in "unit: add config handling in mboxname tests"
Add conversations database.

It's a very simple database with only one type of record, which
maps a message-id with angle brackets, e.g. "<0001@example.com>",
to a conversation id and a timestamp.  The conversation id is a
64b int.

struct conversations_state holds a reference to an open conversations
database.  There can be more than one of these in existance.  There
will eventually be a conversations databases per-user, but this code
doesn't know that and just takes a filename.

conversations_open() opens a database handle and starts a transaction.

conversations_close() aborts the transation if it has not been
    commited, and closes the database handle.

conversations_commit() commits a transaction.

conversations_set() stores a conversation id for a msgid, and sets
    the record's timestamp to the current system time.

conversations_get() retrieves the conversation id given a msgid.

conversations_prune() removes all records older than a given time.

... mega squash commits below

Merge summary:

    merged in "Add conversations_getpath()"
    merged in some hunks of "Add conversationsdump utility."
    merged in some hunks of "Add an imap option to enable conversations"
    merged in some hunks of "Add conversation_id_{de,en}code()"
    merged in some hunks of "Fix compile warnings in conversations code."
    merged in "More fixup for conversion to times.c in imapurl.c"
    merged in some hunks of "conversations_prune() returns statistics"
    merged in some hunks of "Rename conversations_{get,set} -> _{get,set}_cid"
    merged in "Add conversations_add_folder()"
    merged in "Add conversations_rename_cid()"
    merged in "Dump and undump API for conversations."
    merged in some hunks of "conversations: convert to human readable DB format"
    merged in "conversations: use mboxname_conf_getpath"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in some hunks of "conversations: rename get/set_cid to get/set_msgid for clarity"
    merged in "conversations: remove bogus buffer"
    merged in some hunks of "convert conversations to use dlist and support status data, part 1"
    merged in some hunks of "conversations_pass a mailbox object down, ouch"
    merged in "cleanups"
    merged in "remove UNSEEN from perl-folder, we don't care"
    merged in some hunks of "Fix recent conversations API maldesign"
    merged in "Stuff"
    merged in "conversations: don't go segfaulting if starting from scratch"
    merged in "conversations: simplify accounting"
    merged in "conversations: don't fail if cid == 0"
    merged in some hunks of "Some more conversations API rejigging"
    merged in some hunks of "conversations: add sender support"
    merged in "conversations: status API"
    merged in "conversations: gnb review - don't leak keys"
    merged in "conversations: gnb review - sender handling safety"
    merged in "conversations: gnb review - free sender information"
    merged in "conversations: return empty for corrupted data"
    merged in "conversations: don't let counts go negative"
    merged in some hunks of "conversation: add flagged and stub for attachments"
    merged in "conversations: add version information to the front of every record"
    merged in some hunks of "conversation: change "SENDER" format to be the same as ENVELOPE"
    merged in "conversations: fix route fail"
    merged in "conversations: don't delete zero-exists conversations"
    merged in "IRIS 242 - fix XCONVUNSEEN counting"
    merged in "remove debugging line"
    merged in some hunks of "conversations: use cstate from the mailbox for rename callback"
    merged in "conversations: make "NAME" optional in email addresses"
    merged in some hunks of "Fix recent regressions in CID renaming."
    merged in some hunks from "conversations: add XCONVMODSEQ status item to folders"
    merged in "conversations: allow specifying only some fields to getstatus"
    merged in "make conversation_free NULL-safe"
    merged in some hunks of "first rewrite of conversations open and close"
    merged in some hunks of "ctl_converstationsdb - add '-R' option"
    merged in "ctl_conversationsdb cleanup"
    merged in "conversations: fix remove (found by unit tests)"
    merged in "conversations_rename_folder API first draft"
    merged in some hunks of "conversations_rename_folder part 2 - use it."
    merged in some hunks of "conversations_rename_folder: use NULL to_name for delete"
    merged in "remove the 'exists' count for the being-deleted mailbox as well."
    merged in some lines of "Fix parallel build."
    merged in "conversations: fix memory leak"
    merged in some hunks of "conversations: counts per flag (including to-backport dlist usage fixes)"
    merged in some hunks of "gnb review: convert to strarray_t * and clean up"
    merged in "options: document conversations_counted_flags better"
    merged in some hunks of "conversations: remove limit on the number of counted_flags"
    merged in some hunks of "conversations: store counted flags in the DB"
    merged in "conversations: re-init counted flags on a wipe"
    merged in "Fix memleak of conversation_t.counts"
    plus manual cleanups

Add unit test for conversations db.

Merge summary:

    merged in "unit: test conversation_id_encode et al"
    merged in some hunks of "conversations_prune() returns statistics"
    merged in some hunks of "Rename conversations_{get,set} -> _{get,set}_cid"
    merged in "unit: test conversations_add_folder()"
    merged in "unit: test conversations_rename_cid()"
    merged in "unit: test conversations dump API"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in some hunks of "convert conversations to use dlist and support status data, part 1"
    merged in "unit: remove stray printf."
    merged in "cunit: re-enable slow "prune" test"
    merged in some hunks of "Fix recent conversations API maldesign"
    merged in some hunks of "Some more conversations API rejigging"
    merged in some hunks of "unit: update conversation:cid_rename"
    merged in "unit: track conversations_update() changes"
    merged in "unit: track changes to conversations_rename_cid()"
    merged in "cunit: make test work with new conversations API"
    merged in "cunit: test conversations_folder_rename"
    merged in "unit: fix compile warnings"
    merged in "unit: test counted flags"
    merged in "cunit: support counted_flags in DB"
    merged in "unit: fix memleak in conversations tests"

APPEND updates the conversations db

Merge summary:

    merged in "Update conversations for COPY"
    merged in some hunks of "Add an imap option to enable conversations"
    merged in some hunks of "Fix compile warnings in conversations code."
    merged in "Move conversations update code to message.c"
    merged in "Fix memory leak in APPEND command"
    merged in some hunks of "Rename conversations_{get,set} -> _{get,set}_cid"
    merged in some hunks of "Handle CID clashing in APPEND, COPY, LMTP"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in some hunks of "conversations: rename get/set_cid to get/set_msgid for clarity"
    merged in some hunks of "convert conversations to use dlist and support status data, part 1"
    merged in "well, that was dumb"
    merged in some hunks of "conversations_pass a mailbox object down, ouch"
    merged in "two cid updates if cid changes"
    merged in "Clean up mailbox_{open,commit}_conversations"
    merged in some hunks of "Fix recent conversations API maldesign"
    merged in some hunks of "Some more conversations API rejigging"
    merged in some hunks of "conversations: add sender support"
    merged in "conversations: drafts don't count towards unseen"
    merged in some hunks of "conversation: add flagged and stub for attachments"
    merged in "message: simple has_attachment heuristic to give a count"
    merged in "conversations: generate CID even if no message-id headers"
    merged in some hunks of "conversation: change "SENDER" format to be the same as ENVELOPE"
    merged in "conversation: initialise return code from generate"
    merged in "conversations: make "NAME" optional in email addresses"
    merged in "Fix compile warning in message_has_attachment"
    merged in some hunks of "conversations: convert to human readable DB format"
    merged in some hunks of "Fix recent regressions in CID renaming."
    merged in some hunks of "first rewrite of conversations open and close"
    merged in "The easy fix: always lock the conversations db"
    merged in some hunks of "ctl_converstationsdb - add '-R' option"
    merged in "conversations: unlock on mailbox unlock"
    merged in "mailbox: lock conversations on mailbox create too"
    merged in "ctl_conversationsdb: actually, expunged messages exist too"
    merged in "[IRIS-293] Handle CID clash by re-inserting"
    merged in "[IRIS-293] brong review: don't unlink on error."
    merged in some hunks of "conversations: counts per flag (including to-backport dlist usage fixes)"
    merged in some hunks of "gnb review: convert to strarray_t * and clean up"
    merged in some hunks of "conversations: remove limit on the number of counted_flags"
    merged in some hunks of "conversations: store counted flags in the DB"
    merged in "pushback to 0bee8026"
    merged in "pushback to f4203f08"
    manually tweaked to remove message_has_attachment() which has no callers
    manually tweaked for other merge breakage

Conflicts:

	imap/mailbox.c

mailbox fixup

Add conversationsdump utility.

Simple commandline utility to dump the contents of a userid's
conversations database.  Useful for debugging.

Merge summary:

    split off hunks from "Add conversationsdump utility."
    merged in "Fix another compile warning in conversations code"
    merged in "Rename conversationsdump -> ctl_conversationsdb"
    merged in "Add a manpage for ctl_conversationsdb."
    merged in "Add undump option to ctl_conversationsdb"
    merged in "gitignore: change name to ctl_conversationsdb"
    merged in "ctl_conversationsdb: take a userid, not an mboxname"
    merged in "ctl_conversationsdb: zero db and build options"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in "ctl_conversationsdb: factor out do_user"
    merged in "ctl_converstaionsdb: recursive mode"
    merged in some hunks of "first rewrite of conversations open and close"
    merged in some hunks of "ctl_converstationsdb - add '-R' option"
    merged in "ctl_conversationsdb - print mailbox name on recalc if verbose"
    merged in some hunks of "ctl_conversationsdb: actually, expunged messages exist too"
    merged in some lines of "Fix parallel build."

Add a new CID atom to the FETCH command.

Returns "CID" and either

- the conversation id formatted as 16 bytes of hex,
- or "NIL" if the CID in the index is NULLCONVERSATION.

Merge summary:

    merged in some hunks of "Add an imap option to enable conversations"
    merged in some hunks of "Add conversation_id_{de,en}code()"
    merged in some hunks of "conversations: convert to human readable DB format"
    merged in some hunks of "[IRIS-276] fix cleanup on bad cid in XCONVFETCH"
    merged in "Don't emit NIL for non-zero cids."

Sync conversation IDs.

 - The SYNC_CRC_COVERS capability/option supports a new flag "CID"
   which indicates that the CRC covers the CIDs in the index.

 - Sync_server advertises COVERS=CID in the capabilities banner,
   if the "conversations: on" has been specified in imapd.conf.

 - Sync_client passes CIDs with the per-message data in APPLY
   MAILBOX, if "conversations: on" has been specified in imapd.conf.

 - Full update and incremental update for an index record will merge
   the CIDs from the master and replica, choosing the maximum CID
   of the two if they differ and are non-NULL.  During incremental
   update on the server, if this choice would result in the client
   being confused about the CID, throw CRC failure back to force a
   full update so that the client can make the same decision.

 - The case where master and replica have different non-NULL CIDs
   is not handled.  Currently we syslog a whiny message and allow
   the two sides to be inconsistent.  We *should* do a conversation
   merge, rewriting CIDs in the conversations db, but that code
   hasn't been written yet.

Merge summary:

    merged in "sync: fix crc negotiation"
    merged in some hunks of "conversations: convert to human readable DB format"
    merged in some hunks of "Handle CID clashing in APPEND, COPY, LMTP"
    merged in "4a6eb22 wip"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in "SYNC_CHECKSUM"
    merged in "sync: don't care about CID errors on expugned messages"
    merged in some hunks of "conversations: use cstate from the mailbox for rename callback"
    merged in "sync_server: don't throw CID related errors, just pile them up for the end"
    merged in some hunks of "Fix recent regressions in CID renaming."
    merged in some hunks of "first rewrite of conversations open and close"
    merged in "sync_support: free body"
    manually tweaked for merge breakage

cyr_expire expires conversations db entries.

Merge summary:

    merged in some hunks of "first rewrite of conversations open and close"

index: add 'uidvalidity_is_max' flag for xconvfetch

Add FOLDER, UIDVALIDITY fetch items

Merge summary:

    split out from "first pass at XCONVFETCH"
    merged in "Report external name for FOLDER fetch item."
    manually tweaked for merge breakage

xconvmeta: just fiddling around

Merge summary:

    merged in "xconvmeta: rewrite to separate support function, add ATTACHMENTS"
    merged in "xconvmeta - don't break xconvfetch!"
    merged in "xconvmeta: parse in the function"
    merged in some hunks of "conversation: change "SENDER" format to be the same as ENVELOPE"
    merged in some hunks of "xconvmeta: support single items as well as lists"
    merged in some hunks of "xconvmeta: add folderexists"
    merged in "xconvmeta: factor out the response part - only respond if "EXISTS""
    merged in "Another one :("
    merged in some hunks of "first rewrite of conversations open and close"
    merged in some hunks of "conversations: counts per flag (including to-backport dlist usage fixes)"
    merged in some hunks of "gnb review: convert to strarray_t * and clean up"
    merged in some hunks of "conversations: store counted flags in the DB"
    merged in "imapd: remove unused variable"

first pass at XCONVFETCH

Merge summary:

    merged in "Next steps in evolution of XCONVFETCH"
    merged in "XCONVFETCH always returns FOLDER & UIDVALIDITY"
    merged in "New simpler FOLDERSTATE format."
    merged in "Emit empty FOLDERSTATE for missing folders"
    merged in "Use getmodseq()"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in some hunks of "convert conversations to use dlist and support status data, part 1"
    merged in "Fix recent conversations API maldesign"
    merged in "imapd: remove stray variable"
    merged in some hunks of "Some more conversations API rejigging"
    merged in "XCONVFETCH doesn't require a SELECTed mailbox"
    merged in "Simplify XCONVFETCH validity information."
    merged in some hunks of "[IRIS-276] fix cleanup on bad cid in XCONVFETCH"
    merged in "[IRIS-274] Feed index_open properly in XCONVFETCH"
    merged in "xconvfetch: more efficient part 1"
    merged in "xconvfetch: efficiency level 2, skip out if unchanged"
    merged in "xconvfetch: remember to set "out""
    merged in "xconvfetch: provide sender data"
    merged in "xconvfetch: remove SENDERS"
    merged in "xconvfetch: set uidvalidity_is_max"
    merged in "xconvfetch: open index in exclusive mode, so we can do CID renames first"
    merged in "xconvfetch: remove another bogus shortcircuit"
    merged in "xconvfetch: make flags print"
    merged in "XCONVFETCH: support multiple CID in an array"
    merged in "xconvfetch: use dlist_parse_asatomlist as well"
    merged in "xconvfetch: handle mailbox deletion more nicely"
    merged in "xconvfetch: only mention missing folders if the client might care"
    merged in "xconvfetch: use strarray, don't search empty folders if no uidvalidity"
    merged in some hunks of "xconvmeta: add folderexists"
    merged in "use strarray properly"
    merged in "gnb review: remove virtualconv"
    merged in "xconvfetch rewrite - to respond how the frontend API now wants it"
    merged in "xconvfetch: look in folder even if not changed"
    merged in "xconvfetch: don't crash on missing conversation"
    merged in some hunks of "first rewrite of conversations open and close"
    merged in "xconvfetch: lock conversations_db against race condition"
    merged in "xconvfetch: use fetchargs_fini"
    merged in some hunks of "conversations: store counted flags in the DB"
    merged in "pushback to 50099bbf"
    merged in "pushback to 605b3799"
    manually tweaked for minor merge breakage

First cut at XCONVSORT.

Simple form of the syntax is identical to the SORT command.  Extended
form allows adding a (parenthesised) list of windowing arguments between
the sort arguments and the charset.  Currently the only windowing
argument is CONVERSATIONS which indicates that only the ID of the first
message to be encountered from each conversation is to be returned.

Merge history:

    merged in "Add LIMIT to XCONVSORT."
    merged in "Add POSITION, ANCHOR, OFFSET window args"
    merged in "sortcrit_as_string for xconvsort"
    merged in "wip: xconvsort + changedsince"
    merged in "Tighter semantics for XCONVSORT's CHANGED list"
    merged in "Add UNTIL to XCONVSORT"
    merged in "Fix compile warning."
    merged in "[IRIS-275] Fix XCONVSORT on self-expunged messages"
    merged in "[IRIS-275] Fix XCONVSORT 1:*"
    merged in "index: if want_expunged, make sure the index array is big enough"
    merged in "[IRIS-289] Fix no output for XCONVSORT OFFSET < -1"
    merged in "xconvsort: make until take a param as well"
    merged in "xconvsort: regularise sort args"
    merged in "xconvsort: shortcut if folder unchanged"
    merged in "xconvsort: detect conversation changes in other folders"
    merged in "xconvsort: give highestmodseq of all conversations if available"
    merged in "xconvsort: always use xconvmodseq"
    merged in "xconvsort - take a mailboxname, don't require selected"
    merged in "xconvsort - return uidvalidity"
    merged in "xconvsort: report numresults"
    merged in "xconvsort: only look at expunged if changedsince (regression fix)"
    merged in "xconvsort: error if mailbox doesn't exists"
    merged in some hunks of "hash table initializer"
    merged in "xconvsort: add back in is_changed for non-conversations-mode"
    merged in "xconvsort: don't return uninit nmsg"
    merged in "Revert "xconvsort: make until take a param as well""
    merged in "Revert "xconvsort: regularise sort args""
    merged in some hunks of "first rewrite of conversations open and close"
    merged in "convsort: return zero on error"
    merged in "xconvsort: open conversations DB"
    merged in "[IRIS-387] Make NUMRESULTS in XCONVSORT orthogonal"
    merged in "xconvfetch: same 'changedsince' logic for non-conversation path" [ yes really ]
    merged in "Handle SORT_UID in sortcrit_as_string"
    merged in "index_sort rubbish: bloody thing keeps leaking back in"
    manually tweaked to fix merge breakage

imapd: add XCONV* STATUS fields

Merge summary:

    merged in some hunks from "conversations: add XCONVMODSEQ status item to folders"
    merged in "status: make xconvmodseq actually work"
    merged in "xconvmodseq: and also have the correct value"
    merged in "xconvmodseq: use correct format specifier"
    merged in some hunks of "first rewrite of conversations open and close"
    manually tweaked to remove unused variable fname in imapd_statusdata()

conversations_rename_folder part 2 - use it.

Merge summary:

    merged in "Move the conversations db rename AFTER all folder renames"
    merged in some hunks of "conversations_rename_folder: use NULL to_name for delete"
    merged in "rename: don't keep counts if destination is in the DELETED namespace."
    merged in "conversations: don't crash if conversations aren't enabled"
    merged in "user: don't rename conversations if they're not enabled"
    merged in "user: delete conversations and modseq files on delete"
    merged in "Fix a couple of spurious compile warnings."
    manually tweaked for whitespace

Add the "XCONVERSATIONS" capability.

The presence of this post-auth capability indicates that:

- conversations databases are enabled
- the "CID" atom works with the "FETCH" command
- "APPEND", "COPY", and local delivery update the conversations database

[IRIS-204] Parse X-ME-Message-ID for message-ids

and use those to link conversations.

[IRIS-204] unit test parsing X-ME-Message-ID

[IRIS-589] annotation /vendor/cmu/cyrus-imapd/usermodseq

[IRIS-525] xconvsort - support mutable filters and sorts

ctl_conversationsdb - fix up "unused variable"

Realistically, we should use those values for something.

conversations: add "num_records" counter

need to actually check result

conversations: folder_number support

conversations: use offset keys, not names

user: clean up conversations files even if conversations disabled

user: use copyfile to rename conversations db

conversations: fix recently introduced memleaks

imapd: protect xconvsort from conversations: no

imapd: protect STATUS from conversations: no

imapd: protect getsearchcriteria against conversations: no

imapd: protect getsortcriteria against converstaions: no

conversations: copy key for rewrite

Otherwise the mmap offsets may no longer be valid by the time we
get to writing the final record.

mailbox: sync_log mailbox on any "post action"

This will force conversation renames to finish quickly, and hence
be pushed correctly to the replica.

conversations: don't rename_cid on the replica end of any transfer

IRIS-293 - revert CID rename reinjection: it breaks IOS

conv rename fix

conversations: only rename cids on the master

This stops modseq bounces causing problems on the replica end.

Basically, the master should do all the work... and the replica
just slavishly copy it.

[IRIS-893] Fix COPY between users

The wrong conversation database was being opened in append_copy(), i.e.
the db for the source folder not the destination folder.  For copying
between any two folders of the same user, this doesn't matter, but it
breaks when copying between folders of different users.

This originally worked but regressed during one of the conversation API
rewrites.

Also, fix an incorrect comment for mboxname_parts_same_userid(); it
cannot return a negative error.

Move index_pruneheader() to message.c

and rename it message_pruneheader().  It has nothing whatsoever to do
with the index.c structures, but the actual reason is that I'm about to
need to call it from message.c and some of the cyrus utilities don't
link if message.o needs index.o.

Cache X-ME-Message-ID in cyrus.cache

so that we can use it soon for updating conversations from the cache.

Update conversations from cache if body = NULL

We (now) have enough information from the cyrus.cache to be able
to update the conversation threading correctly for a given message,
so the 'body' parameter to message_update_conversations() is now
optional.

Remove message_update_conversations_file()

which was the wrapper for message_update_conversations() which parsed
the whole message into a struct body, for the two callsites which needed
to update conversations but didn't have a struct body.  One of these was
the APPEND path and the other was ctl_conversationsdb -b.  As it
happens, both of those have cyrus.cache information available so we can
just call message_update_conversations() with a NULL body*.

ctl_conversationsdb: don't process users with DELETED mailboxes twice

ctl_conversationsdb - log sync events

mailbox: move the cstate fetch to where it actually matters

ctl_conversationsdb: build CID for expunged messages too

conversations: use size_t for DB access

[IRIS-237] Implement XCONVSORT v2

New command design, illuminated by experience.

 - split XCONVSORT command into XCONVSORT and XCONVUPDATES where
   XCONVUPDATES has the CHANGEDSINCE window-args.

 - limit the OFFSET window-arg to be non-negative

 - the server returns an error if an ANCHOR is not found, forcing the
   client to retry with POSITION, rather than trying to implement
   both with a fallback.

This combination of design changes makes the implementation in index.c
much simpler, and a vast amount of complexity disappears (including an
entire pass).  Also a number of useful minor design tweaks:

 - window-args format changed so that any arguments for an item
   are in (brackets) which makes it a whole lot easier to generate
   or parse in Perl

 - rejigged ANCHOR, OFFSET, POSITION and LIMIT items into a more
   sensible set of combinations

 - XCONVUPDATES has an UPTO window-args item which is like ANCHOR+UNTIL
   but differs subtly in failure modes.  Remove UNTIL

 - Rejigged the order in which the 'charset' argument appears so that
   the ABNF makes more sense

Fix memleak in XCONVUPDATES

introduced in "[IRIS-237] Implement XCONVSORT v2"

Fix TOTAL for XCONVSORT...NOT SEEN

Note that 'UNSEEN' worked, but 'NOT SEEN' and 'NOT UNSEEN'
were broken and reported the count of the opposite condition.

Don't optimise TOTAL for XCONVSORT..RECENT

we don't use it and it's too hard to test.  Just always fall back
to the slow but accurate case.

Add conversation_get_modseq

which returns just the modseq of a CID from the B record using a very
simple and fast parse which ignores most of the record.  Profiling
showed that with large mailboxes parsing the B record as a dlist was
actually the dominant cost of an xconvupdates call.

Size CID hash tables properly in xconvupdates

to avoid excessive chaining on large mailboxes.  Kcachegrind shows distribution
to callees of cycles spent in index_convupdates, before:

index_convupdates
	26.08 lsort
	23.69 get_modseq_of		# called 25,202 times
	20.04 hash_lookup		# called 221,307 times
	10.95 conversation_id_encode	# called 110,654 times
	 7.72 hash_insert		# called 50,405 times
	 4.21 index_search_evaluate	# called 110,654
	 2.55 index_msgdata_free	# called 110,654
	 1.94 index_msgdata_load	# called 1
	 1.15 free_hash_table		# called 2

and after:

index_convupdates
	30.84 lsort
	28.00 get_modseq_of
	12.94 conversation_id_encode
	 9.16 hash_lookup
	 5.30 hash_insert
	 4.97 index_search_evaluate
	 3.01 index_msgdata_free
	 2.30 index_msgdata_load

There's still significant room for improvement in the hashing.

Use qsort for SORT,THREAD,XCONVSORT,XCONVUPDATES

instead of lsort - experiment indicates it's about twice as fast for
large mailboxes.  From callgrind reports of percentage distribution to
callees of cpu cycles, before:

index_convupdates
	30.84 lsort		<----
	28.00 get_modseq_of
	12.94 conversation_id_encode
	 9.16 hash_lookup
	 5.30 hash_insert
	 4.97 index_search_evaluate
	 3.01 index_msgdata_free
	 2.30 index_msgdata_load

and after:

index_convupdates
	32.07 get_modseq_of
	20.46 qsort		<----
	14.82 conversation_id_encode
	10.49 hash_lookup
	 6.07 hash_insert
	 5.69 index_search_evaluate
	 3.45 index_msgdata_free
	 1.70 free_hash_table

In absolute terms, lsort used 219,619,238 of however callgrind counts
cycles and qsort 127,211,813.  This was for a mailbox of 110,654
messages.

XCONVUPDATES uses a hash table keyed on uint64_t

instead of strings.  This saves a significant amount of time on large
mailboxes in conversations mode.  From callgrind reports of percentage
distribution to callees of cpu cycles, before:

index_convupdates
	32.07 get_modseq_of
	20.46 qsort
	14.82 conversation_id_encode	    <----
	10.49 hash_lookup		    <----
	 6.07 hash_insert		    <----
	 5.69 index_search_evaluate
	 3.45 index_msgdata_free
	 1.70 free_hash_table

and after:

index_convupdates
	41.19 get_modseq_of
	34.10 qsort
	 7.31 index_search_evaluate
	 4.43 index_msgdata_free
	 3.42 index_msgdata_load
	 2.81 hashu64_insert		<----
	 2.68 hashu64_lookup		<----
	 1.24 free_hashu64_table

[IRIS-1072] Fix XCONVUPDATES crash on empty folder

It turns out you can't initialise a hashtable with zero size.  Woops.

[IRIS-1080] Remove mboxname from XCONVSORT/UPDATES

As it's no longer useful and is inconsistent with existing IMAP
commands, per RobM.

Predict correct TOTAL for XCONVUPDATES...ALL

when any expunged messages are present in the mailbox.  It turns out
that the index_state->want_expunged hack had the unfortunate side effect
of subtly changing the semantics of index_state->exists.  Worked around
by adding index_state->num_expunged which is updated in index_refresh()
but only if ->want_expunged is true (i.e. only for XCONVUPDATES).

[IRIS-1076] Another crash on empty folder

Index_msgdata_free() couldn't handle the case of a NULL msgdata* but a
non-zero count, which could happen in XCONVSORT and XCONVUPDATES if the
search total predictor reported a zero result (e.g. an empty folder).

Also, for XCONVUPDATES, we can't short circuit the main loop if the
mailbox currently has no messages but there are expunged messages which
might need to be reported as REMOVED.

[IRIS-1080] handle failure while alternate index is open

If we're doing updates, we need to create a new imapd_index,
and if we then fail, we need to close that index and make
sure we're looking at the original again.

conversation_load can return true but no data, so cope

conversations: return real values for modseq

size_t changes for conversations as well

[IRIS-975] Be more pedantic about In-Reply-To:

message_update_conversations: never promote the CID if we're a replica

sync: just update this one message if changing CID

sync: really, whoever has the highest modseq wins for cid

conversations: clean out when NUMRECORDS == 0

fixup comment about cache records

twoskip: all for conversations db too

Refactor message_update_conversations

to separate out the CID update logic and the database reading & writing
logic a bit more clearly, but more importantly make the effects of the
isreplica flag a lot easier to understand.

rename_cid: don't rename if it's the same!

fix up conversations DB usage

conversations: don't break the DB contract

conversations: don't assert on EXPUNGED records in index

[IRIS-1144] expunge messages during convsort

re-add cleanup of uidvalidity - for backport

conversations: static some more

conversations: remove all records when deleting

Otherwise the counts will be wrong

conversations: suppress storing for DELETED mailboxes

unit: fix annotate tests for quota

They now need the quotadb open, and existing mailbox on-disk structures.

ctl_conversationsdb: add '-A' flag to audit counts

remove modseq bogons

audit: don't count unseen if it doesn't even exist

cstate: always commit, not abort, after xconvsort

ctl_conversationsdb: also commit where it doesn't matter

ctl_conversationsdb: use error_message()

to print more information about why something failed.  Also, remove the
comment spuriously complaining about the lack of something like
error_message().

[IRIS-30] update ctl_conversationsdb documentation

Update to show the current syntax, which specifies a userid and not a
mailbox.  Document the -z -b and -R options for doing reconstruction.
Document the -r option for recursion.  Document that combining -u and -r
is probably a really bad idea.  General cleanup.

Fix compile warning in ctl_conversationsdb

[IRIS-1073] be less crashy if conversations db bad

Fix a memory leak in cyr_expire

mbexamine prints the CID for each message

Add mailbox_get_xconvmodseq()

and mailbox_update_xconvmodseq().  These functions pretend that some of
the state stored in the conversations database is a property of the
mailbox (logically, it really is).  They require the mailbox to have
been opened either writably or with MBFLAG_CONVERSATIONS, and return
IMAP_INTERNAL if this is not the case.

[IRIS-1182] Replicate folder XCONVMODSEQ

as if it were a first class property of the mailbox, rather than
a side effect of message appends.

xconvmodseq sync - just use _iwl instead for now

server: abort if xconvmodseq would go down

server fixup

replicate changes back too

syslog error_message(r) per gnb code review

conversations: add conversations_set_suffix()

which allows the caller to change the way in which the pathnames of
conversations dbs are created from mboxnames.  Normally this is not
useful, but for conversation db audit we need to make sure that existing
mailbox code will do things into a special temporary db instead of the
live one.

[IRIS-1287] conversations: replace audit mode

Rewrote the code that implements ctl_conversationdb -A.  Now it:

 *  copies the live conversations db to a temp copy
 *  does a wipe & recalculate on the temp copy
 *  opens both live and temp copies
 *  scans through them comparing records and complaining if they differ
 *  if there were differences, tells you on stdout
 *  if -v is specified, tells you the differences in detail
 *  unlinks the temp copy

Note that we deliberately keep a lock on the user for the shortest
possible time.  The upside is that normal Cyrus operation is not held
out for long.  The downside is that the audit is racy w.r.t. new
messages arriving in the user's mailboxes, which may trigger a spurious
report of inconsistency.

conversations: sort folder names

cunit: test folder ordering

conversations: sort senders too

cunit: test conversations sender ordering

conversations: add senders even if expunged

conversations: audit internal counts on load and save

[IRIS-1330] Don't break conversations db on RENAME

At the tail end of the RENAME command, mailbox_rename_cleanup() was
calling mailbox_delete() on the old copy of the folder which had just
been renamed.  Mailbox_delete() updated the conversations db as if the
folder had been deleted, bumping down exists counts etc.  Unfortunately,
the exists counts had *not* been bumped up when the folder was copied
earlier in the rename process; instead conversation_rename_folder() had
rewritten the B records with the same counts but the new folder name.

The effect was to at least break the counts on every B record touched by
the folder.  Conversations existing only in that one folder had their B
record deleted.

The quick and ugly solution is to pass an extra parameter to
mailbox_delete().  Better solutions are invited!

cunit: fix CID-related compile warnings on 32b

cunit: fix memleak in conversations tests

conversations: change num_records in folder delete too

conversations: wipe - delete with correct name length

conversations: use CONV_FMT for syslogs

conversations: rename the $FOLDER_NAMES entry instead

Rather than complex renaming magic, just rename the one entry!
This is heaps more efficient, simpler, less error prone :)

And it's going to look more like how it would be after a
recreate too...

Except the ordering will be wrong in all the records, ho hum.
I think we should be sorting them by $FOLDER_NAMES entry rather
than by name, except then it depends how THEY are allocated.

conversations: sort by folder_number, not folder_name

We do this by not wiping the folder number on rebuild.

This allows us to have reliable reuse of folder names.  It does mean
that if you create and delete thousands of folders, you will get a
giant record - but that's life.

On the plus side, it means that B records should actually be BYTE
identical, not just "parsed foldernames" identical.

conversations update: failure error message

conversations: don't break on deleting an empty folder

conversations: handle empty folder for key too

conversations: don't syslog NULL to_name

conversations: break out of get_folder loop early

[IRIS-723] Use X-ME-Message-ID for duplicates

The contens of the header field X-ME-Message-ID is used as a message's
id for duplicate suppression purposes if the header is present.
Otherwise we fall back to Resent-Message-ID and Message-ID.

[IRIS-1373] auto EXPUNGE for XCONV{UPDATES,FETCH}

This ensures all messages flagged \Deleted are expunged before
running any of the XCONV* commands which tells us about individual
messages.

cunit fixup for 1399

[IRIS-1399] fix conversation_find_folder

to return either the correct folder or NULL.

Also, don't add unnecessary entries to $FOLDER_NAMES if
conversation_find_folder() is called with a folder name which does not
exist.  This mboxname comes in directly off the wire and could be any
old rubbish, not necessarily a current or future valid folder name.

mailbox: skip records without a CID for updates

remove actions for CID rename

remove unit test for rename, it doesn't cope with not having a real mailbox any more

Add -T option to ctl_conversationsdb

which specifies a directory into which the temporary copy of the
conversations database will be copied, e.g.  a directory mounted on
tmpfs.

conversations - reverse the order of CID updates

rename_cid: don't rename if the cid isn't changed (duh)

rename_cid: use mailbox_findopen to find existing mailboxes

mailbox: assign modseq before doing conversations update in append

rename_cid: ignore if from_cid is zero, and assert it's an increase

conversations: don't crash if deleting a folder with UNLINKED messages

ctl_conversationsdb: audit without losing modseq history

ctl_conversationsdb: document the facts about modseqs properly

[IRIS-1484] Be more liberal uniquifying SENDERS

- mailbox & domain compared case-insensitively

- smarter heuristics to choose best name

- ensure results are consistent regardless of message arrival order, so
  that conversations audit is not confused by reconstruction or
  replication.

ctl_conversationsdb: add SENDERS to the Audit copy too

ctl_conversationsdb: make sure we actually WRITE the changes, sheesh

ctl_conversationsdb - don't try to add UNLINKED records

Fix compile warning in ctl_conversationsdb.c

Fix numerous warnings in unit tests

These things just keep on creeping back.  One of these days I'm going to
add -Werror into CFLAGS by default...

Add conversations_normalise_subject()

which takes a MIME-decoded subject string and normalises it in-place.

Add conversations_{get,set}_subject()

which uses a new 'S' record in the conversations DB to store the
normalised subject string of a conversation.

This could have gone into the 'B' record as a new optional field or
even a version number bump; I'm still not 100% convinced this was the
best solution.

adding back bits

Non ME stuff

[IRIS-1466] new conversation when subject changes

when choosing a conversation id for a newly arrived message, don't pick
a conversation whose normalised subject differs from the message's.

conversations: don't crash on missing subject

conversations: abort if a count tries to go below zero

conversations_wipe: clean up 'S' keys (IRIS-1466) and $FOLDER_NAMES

conversations: but keep $FOLDER_NAMES if auditing

ctl_conversationsdb: add '-F' go check folder_names for duplicates

ctl_conversationsdb: -F also print OK if they match

conversations: don't segfault on audit

wipe_counts - choose if names are kept

ctl_conversationsdb: don't infinite loop cursor if DBs don't have the same final key

short-circuit if one side has already finished

[IRIS-1466] parse Subject header (from gnb)

ctl_conversationsdb - handle missing records better everywhere

remove unused 'i'

cursor - wow, this really sucks

ctl_conversationsdb: -z, just zero CIDs, no other cleanup

remove 'S' key when last record is deleted

conversations: reuse delete folder name slots in $FOLDER_NAMES

This is to stop $FOLDER_NAMES becoming a huge pile of '-' in
the degenerate case

talk about your "lost commits" - fix strarray_find

[IRIS-1466] match NULL/empty subject correctly

A NULL or empty Subject header field in a message does not match any
non-empty subject in an S record, it only matches empty subjects or
subjects which normalise to empty such as "Re:".

We still handle a missing S-record as a special match-anything case
which indicates that the conversations DB has not been reconstructed
since the IRIS1466 code was deployed, and which therefore matches any
incoming subject.

conversations: only user folders have conversations at all

ctl_converstaionsdb - fix and clarify source info in Audit

ctl_conversationsdb: fix modseq checking on folders

ctl_conversationsdb: don't warn on missing SUBJECT

conversations: sort by encoded form if idential decoded

conversations: tidy up next_diffable

cyr_expire: some folders have no conversations DB

Be consistent about detecting bad conv db records

Always return IMAP_MAILBOX_BADFORMAT instead of some combination of 0,
IMAP_MAILBOX_BADFORMAT, and IMAP_INTERNAL.  In particular, if
conversation_parse() returns 0 then it has successfully parsed a
conversation and the conversation_t* is a non-NULL.

The Conversations.db_audit test in Cassandane deliberately triggers this
failure mode by damaging a DB record directly using cyr_dbtool.
Code in the audit path, recently added to fix up SENDERS information,
was crashing because conversation_parse() was returning 0 and a NULL
conversation.

Don't stop early when auditing corrupted DBs

If a conversations DB has a single corrupted record, do not stop the
audit early but continue to detect problems with other records.  We only
stop early if we encounter an error writing to the temporary DB, and
then we clearly say "giving up".

Also, print useful error messages when encountering failures in
fix_modseqs(), including the DB pathname and the record key.

[IRIS-1576] X-ME-Message-ID overrides Subject

If X-ME-Message-ID says the messages are linked, ignore any difference
in the normalised Subject: header fields.

ctl_conversationsdb: free conversations during fix_modseq

conversations_update: be safer and simpler about UNLINKED

mailbox: don't crash when naming FROM no state TO something with state

annotate: remove begin

fix up API usage from master changes

don't crash on delete of already gone conversation (from gnb)

[TEMPFIX] just zero out invalid conversation deltas so we don't crash

[IRIS-1572] allow quoting charset arg to XCONVSORT

Fix comment typo in xconvupdates

[IRIS-1573] don't ignore XCONVUPDATES UPTO param

Whitespace fixes in cyr_expire.c.

refcount senders - somewhat backwards compatible

ctl_conversationsdb: senders no longer need to be copied

This is because they're now reference counted, so stale
senders won't be around any more.

move conversation parsing into mailbox (part 1/2)

The second part will involve pulling the code out from
message.c into mailbox.c, where it will actually return
the actual 'conversation_t *' that it found when doing
subject lookups, so that mailbox_update_conversations
can just update that one (even if it's brand new, it will
need to have the subject in it already)

conversations: write out updated SENDER state too

conversations: move S keys into B keys

conversations: keep senders sorted in display order in DB

remove debugging, oops

[IRIS-1745] store size in conversations DB

[IRIS-1745] also, "SIZE" option to XCONVMETA

conversations: remove dead "S" key handling code

conversations: don't add sender if count is zero

conversations: explict mailbox_add_conversations

API to update new values into a new conversations database,
and hence handle renames between users (or between different
conversations roots in general)

ctl_conversations: use add_conversations API

conversations: never save more than 100 senders

conversations: sync_log the mailbox on F key write

conversations: remove unused vars

sync: add back xconvmodseq

[IRIS-1944]: don't try to load conversations DB if not in conv mode
[IRIS-1110] Add XWARMUP command

Syntax is, in ABNF

command =/ 'XWARMUP' SP mboxname SP '(' *warmup-items ')

warmup-item = 'INDEX' / 'CONVERSATIONS' / 'ANNOTATIONS'
		/ 'FOLDERSTATUS' / 'ALL'

The command does works whether or not a mailbox is currently SELECTed.
The command requires read permission for the mailbox.  The only result
are OK/NO/BAD, there are no untagged responses.  The command is not
announced in any capability.

[IRIS-1110] use mbentry for XWARMUP

Thanks to code review from RobM, who pointed out that index_open() is
going to read the entire cyrus.index file in blocking mode, which is not
quite what we want to achieve.

Suppress ENOENT errors in index_warmup()

Obviously, it's not an error to be unable to preload a file because it
doesn't exist.  But for some reason we've seen an ENOENT be reported to
syslog in production, so suppress it just before the error message
instead of where I thought it would be returned from a syscall.

[IRIS-1816] Handle posix_fadvise errors right

this time for sure.

[IRIS-1816] Ignore EINVAL from posix_fadvise

why did I ever check the return value.  Why???

[IRIS-1816] Suppress warmup errors properly.

Mondays.  I never could get the hang of Mondays.
Use bitvector_t in squat glue code

which code uses large bit vectors to track whether messages in a mailbox
have matched the search strings or not.
Move Squat glue code into search_squat.c

because we want search_engines.c to be generic routines only.