Bug #3667 - ignore .SILENT if qresync enabled (part 2)

In 2.5.x, we also store annotations per message, and they also
need the .SILENT ignoring
Bug #3651 - fix rehash typos from 2.4.15

Thanks carlos.velasco@nimastelecom.com for finding and patching
these so quickly!
Remove sieve action string

We use sieve to forward messages to a notification
daemon which can send SMSes, and we don't want them
filled with information about the actions which
Sieve took.

This patch stops the action_string being appended to
the message sent to the notify daemon.
GUID IMAP COMMANDS

This patch factors out stuff we used to have in the old MD5UUIDs
patch, the following FETCH responses in imapd:

FETCH DIGEST.SHA1 => 40 character hex string (message sha1)
FETCH RFC822.SHA1 => 40 character hex string (message sha1, calculated)
FETCH RFC822.FILESIZE => size of actual file on disk (via stat or mmap)

It also adds a capability string item: "DIGEST=SHA1"

Totally non-standard of course, but way useful for our replication checking
scripts.  Embrace and extend 'r' us.

Anyone feel like writing an RFC for fetching the digest of a message via
IMAP?  If the server calculated it on delivery and cached it then you'd have
a great way to clean up after a UIDVALIDITY change or other destabilising event
without having to fetch every message again.
Tool to build debian package directory
actually, just not git
Fatal -> abort in common daemons
Fastmail Secrets (no rated)
Add conversations database.

It's a very simple database with only one type of record, which
maps a message-id with angle brackets, e.g. "<0001@example.com>",
to a conversation id and a timestamp.  The conversation id is a
64b int.

struct conversations_state holds a reference to an open conversations
database.  There can be more than one of these in existance.  There
will eventually be a conversations databases per-user, but this code
doesn't know that and just takes a filename.

conversations_open() opens a database handle and starts a transaction.

conversations_close() aborts the transation if it has not been
    commited, and closes the database handle.

conversations_commit() commits a transaction.

conversations_set() stores a conversation id for a msgid, and sets
    the record's timestamp to the current system time.

conversations_get() retrieves the conversation id given a msgid.

conversations_prune() removes all records older than a given time.

Merge summary:

    merged in "Add conversations_getpath()"
    merged in some hunks of "Add conversationsdump utility."
    merged in some hunks of "Add an imap option to enable conversations"
    merged in some hunks of "Add conversation_id_{de,en}code()"
    merged in some hunks of "Fix compile warnings in conversations code."
    merged in "More fixup for conversion to times.c in imapurl.c"
    merged in some hunks of "conversations_prune() returns statistics"
    merged in some hunks of "Rename conversations_{get,set} -> _{get,set}_cid"
    merged in "Add conversations_add_folder()"
    merged in "Add conversations_rename_cid()"
    merged in "Dump and undump API for conversations."
    merged in some hunks of "conversations: convert to human readable DB format"
    merged in "conversations: use mboxname_conf_getpath"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in some hunks of "conversations: rename get/set_cid to get/set_msgid for clarity"
    merged in "conversations: remove bogus buffer"
    merged in some hunks of "convert conversations to use dlist and support status data, part 1"
    merged in some hunks of "conversations_pass a mailbox object down, ouch"
    merged in "cleanups"
    merged in "remove UNSEEN from perl-folder, we don't care"
    merged in some hunks of "Fix recent conversations API maldesign"
    merged in "Stuff"
    merged in "conversations: don't go segfaulting if starting from scratch"
    merged in "conversations: simplify accounting"
    merged in "conversations: don't fail if cid == 0"
    merged in some hunks of "Some more conversations API rejigging"
    merged in some hunks of "conversations: add sender support"
    merged in "conversations: status API"
    merged in "conversations: gnb review - don't leak keys"
    merged in "conversations: gnb review - sender handling safety"
    merged in "conversations: gnb review - free sender information"
    merged in "conversations: return empty for corrupted data"
    merged in "conversations: don't let counts go negative"
    merged in some hunks of "conversation: add flagged and stub for attachments"
    merged in "conversations: add version information to the front of every record"
    merged in some hunks of "conversation: change "SENDER" format to be the same as ENVELOPE"
    merged in "conversations: fix route fail"
    merged in "conversations: don't delete zero-exists conversations"
    merged in "IRIS 242 - fix XCONVUNSEEN counting"
    merged in "remove debugging line"
    merged in some hunks of "conversations: use cstate from the mailbox for rename callback"
    merged in "conversations: make "NAME" optional in email addresses"
    merged in some hunks of "Fix recent regressions in CID renaming."
    merged in some hunks from "conversations: add XCONVMODSEQ status item to folders"
    merged in "conversations: allow specifying only some fields to getstatus"
    merged in "make conversation_free NULL-safe"
    merged in some hunks of "first rewrite of conversations open and close"
    merged in some hunks of "ctl_converstationsdb - add '-R' option"
    merged in "ctl_conversationsdb cleanup"
    merged in "conversations: fix remove (found by unit tests)"
    merged in "conversations_rename_folder API first draft"
    merged in some hunks of "conversations_rename_folder part 2 - use it."
    merged in some hunks of "conversations_rename_folder: use NULL to_name for delete"
    merged in "remove the 'exists' count for the being-deleted mailbox as well."
    merged in some lines of "Fix parallel build."
    merged in "conversations: fix memory leak"
    merged in some hunks of "conversations: counts per flag (including to-backport dlist usage fixes)"
    merged in some hunks of "gnb review: convert to strarray_t * and clean up"
    merged in "options: document conversations_counted_flags better"
    merged in some hunks of "conversations: remove limit on the number of counted_flags"
    merged in some hunks of "conversations: store counted flags in the DB"
    merged in "conversations: re-init counted flags on a wipe"
    merged in "Fix memleak of conversation_t.counts"
    plus manual cleanups
mboxname: add nextmodseq and nextuidvalidity APIs

Merge summary:

    merged in "mboxname: initialise fd"
    merged in "use per-user meta files to serialize uidvalidity and highestmodseq"
    merged in "only use per-user modseq and uidvalidity if conversations are enabled"
    merged in "managed values: add set API and use in replication"
    merged in "mboxname_setval: make sync_server value setting work correctly"
    merged in "sync: handle highestmodseq correctly on mailbox create"
    merged in "cunit: test mboxname_nextmodseq (currently broken)"
    merged in "Fix the mboxname.nextmodseq test."
    merged in "unit: add config handling in mboxname tests"
Add unit test for conversations db.

Merge summary:

    merged in "unit: test conversation_id_encode et al"
    merged in some hunks of "conversations_prune() returns statistics"
    merged in some hunks of "Rename conversations_{get,set} -> _{get,set}_cid"
    merged in "unit: test conversations_add_folder()"
    merged in "unit: test conversations_rename_cid()"
    merged in "unit: test conversations dump API"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in some hunks of "convert conversations to use dlist and support status data, part 1"
    merged in "unit: remove stray printf."
    merged in "cunit: re-enable slow "prune" test"
    merged in some hunks of "Fix recent conversations API maldesign"
    merged in some hunks of "Some more conversations API rejigging"
    merged in some hunks of "unit: update conversation:cid_rename"
    merged in "unit: track conversations_update() changes"
    merged in "unit: track changes to conversations_rename_cid()"
    merged in "cunit: make test work with new conversations API"
    merged in "cunit: test conversations_folder_rename"
    merged in "unit: fix compile warnings"
    merged in "unit: test counted flags"
    merged in "cunit: support counted_flags in DB"
    merged in "unit: fix memleak in conversations tests"
Add mailbox actions file.

This is a file where actions can be posted, that are then later executed
by the first process to open the mailbox with an exclusive lock.  A
fcntl lock is used to provide exclusion between writers to file file,
and to detect when a writer has lost a race with an opener (who renames
the actions file before excuting the actions therein).  Actions are
simple lines of text.

Currently only a single action is implemented, the NOP action which just
increments a count.  This is for testing purposes.

The purpose of the mechanism is to allow delayed renaming of CIDs in
mailbox index files.

Merge summary:

    manually tweaked to add #include "strarray.h"
unit: test mailbox actions posting

Merge summary:

    merged in "mailbox: update unit tests"
    merged in "unit: use config_reset()"
APPEND updates the conversations db

Merge summary:

    merged in "Update conversations for COPY"
    merged in some hunks of "Add an imap option to enable conversations"
    merged in some hunks of "Fix compile warnings in conversations code."
    merged in "Move conversations update code to message.c"
    merged in "Fix memory leak in APPEND command"
    merged in some hunks of "Rename conversations_{get,set} -> _{get,set}_cid"
    merged in some hunks of "Handle CID clashing in APPEND, COPY, LMTP"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in some hunks of "conversations: rename get/set_cid to get/set_msgid for clarity"
    merged in some hunks of "convert conversations to use dlist and support status data, part 1"
    merged in "well, that was dumb"
    merged in some hunks of "conversations_pass a mailbox object down, ouch"
    merged in "two cid updates if cid changes"
    merged in "Clean up mailbox_{open,commit}_conversations"
    merged in some hunks of "Fix recent conversations API maldesign"
    merged in some hunks of "Some more conversations API rejigging"
    merged in some hunks of "conversations: add sender support"
    merged in "conversations: drafts don't count towards unseen"
    merged in some hunks of "conversation: add flagged and stub for attachments"
    merged in "message: simple has_attachment heuristic to give a count"
    merged in "conversations: generate CID even if no message-id headers"
    merged in some hunks of "conversation: change "SENDER" format to be the same as ENVELOPE"
    merged in "conversation: initialise return code from generate"
    merged in "conversations: make "NAME" optional in email addresses"
    merged in "Fix compile warning in message_has_attachment"
    merged in some hunks of "conversations: convert to human readable DB format"
    merged in some hunks of "Fix recent regressions in CID renaming."
    merged in some hunks of "first rewrite of conversations open and close"
    merged in "The easy fix: always lock the conversations db"
    merged in some hunks of "ctl_converstationsdb - add '-R' option"
    merged in "conversations: unlock on mailbox unlock"
    merged in "mailbox: lock conversations on mailbox create too"
    merged in "ctl_conversationsdb: actually, expunged messages exist too"
    merged in "[IRIS-293] Handle CID clash by re-inserting"
    merged in "[IRIS-293] brong review: don't unlink on error."
    merged in some hunks of "conversations: counts per flag (including to-backport dlist usage fixes)"
    merged in some hunks of "gnb review: convert to strarray_t * and clean up"
    merged in some hunks of "conversations: remove limit on the number of counted_flags"
    merged in some hunks of "conversations: store counted flags in the DB"
    merged in "pushback to 0bee8026"
    merged in "pushback to f4203f08"
    manually tweaked to remove message_has_attachment() which has no callers
    manually tweaked for other merge breakage

Conflicts:

	imap/mailbox.c
mailbox fixup
Add conversationsdump utility.

Simple commandline utility to dump the contents of a userid's
conversations database.  Useful for debugging.

Merge summary:

    split off hunks from "Add conversationsdump utility."
    merged in "Fix another compile warning in conversations code"
    merged in "Rename conversationsdump -> ctl_conversationsdb"
    merged in "Add a manpage for ctl_conversationsdb."
    merged in "Add undump option to ctl_conversationsdb"
    merged in "gitignore: change name to ctl_conversationsdb"
    merged in "ctl_conversationsdb: take a userid, not an mboxname"
    merged in "ctl_conversationsdb: zero db and build options"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in "ctl_conversationsdb: factor out do_user"
    merged in "ctl_converstaionsdb: recursive mode"
    merged in some hunks of "first rewrite of conversations open and close"
    merged in some hunks of "ctl_converstationsdb - add '-R' option"
    merged in "ctl_conversationsdb - print mailbox name on recalc if verbose"
    merged in some hunks of "ctl_conversationsdb: actually, expunged messages exist too"
    merged in some lines of "Fix parallel build."
Add a new CID atom to the FETCH command.

Returns "CID" and either

- the conversation id formatted as 16 bytes of hex,
- or "NIL" if the CID in the index is NULLCONVERSATION.

Merge summary:

    merged in some hunks of "Add an imap option to enable conversations"
    merged in some hunks of "Add conversation_id_{de,en}code()"
    merged in some hunks of "conversations: convert to human readable DB format"
    merged in some hunks of "[IRIS-276] fix cleanup on bad cid in XCONVFETCH"
    merged in "Don't emit NIL for non-zero cids."
Sync conversation IDs.

 - The SYNC_CRC_COVERS capability/option supports a new flag "CID"
   which indicates that the CRC covers the CIDs in the index.

 - Sync_server advertises COVERS=CID in the capabilities banner,
   if the "conversations: on" has been specified in imapd.conf.

 - Sync_client passes CIDs with the per-message data in APPLY
   MAILBOX, if "conversations: on" has been specified in imapd.conf.

 - Full update and incremental update for an index record will merge
   the CIDs from the master and replica, choosing the maximum CID
   of the two if they differ and are non-NULL.  During incremental
   update on the server, if this choice would result in the client
   being confused about the CID, throw CRC failure back to force a
   full update so that the client can make the same decision.

 - The case where master and replica have different non-NULL CIDs
   is not handled.  Currently we syslog a whiny message and allow
   the two sides to be inconsistent.  We *should* do a conversation
   merge, rewriting CIDs in the conversations db, but that code
   hasn't been written yet.

Merge summary:

    merged in "sync: fix crc negotiation"
    merged in some hunks of "conversations: convert to human readable DB format"
    merged in some hunks of "Handle CID clashing in APPEND, COPY, LMTP"
    merged in "4a6eb22 wip"
    merged in some hunks of "add highestmodseq to conversations db"
    merged in "SYNC_CHECKSUM"
    merged in "sync: don't care about CID errors on expugned messages"
    merged in some hunks of "conversations: use cstate from the mailbox for rename callback"
    merged in "sync_server: don't throw CID related errors, just pile them up for the end"
    merged in some hunks of "Fix recent regressions in CID renaming."
    merged in some hunks of "first rewrite of conversations open and close"
    merged in "sync_support: free body"
    manually tweaked for merge breakage
cyr_expire expires conversations db entries.

Merge summary:

    merged in some hunks of "first rewrite of conversations open and close"
index: add 'uidvalidity_is_max' flag for xconvfetch