Correctly get default realm in kdb5.c

Call krb5_get_default_realm instead of directly accessing
context->default_realm, to remove the requirement that
krb5_get_default_realm or krb5_set_default_realm be used before
krb5_db_open.
Fix SAM-2 preauth when password argument is used

sam2_process accesses gak_data to get the password, so that it can do
string-to-key with the etype in the SAM-2 challenge.  When #7642
changed gic_pwd.c to use struct gak_password instead of krb5_data,
sam2_process wasn't altered to match.  We don't see a problem when the
password is read through the prompter (as with kinit), because the
password winds up in the storage field at the beginning of the
gak_password structure.  But when a password is supplied as a
parameter (as with ksu), the storage field is empty and we get the
wrong answer from sam2_process.

ticket: 7862
target_version: 1.12.2
tags: pullup
Don't use system dictionary files for DB2 tests

The system dictionary may contain entries with punctuation, which can
confuse the shell.  It's more predictable to always use the word list
from the source tree.

ticket: 7860
status: open
Generate dependencies for t_stringattr

When t_stringattr.c was introduced, it wasn't added to EXTRADEPSRCS,
so no dependencies were created for it.  Fix that now.
Update doc build instructions

Documentation build instructions in doc/README were out of date.
Update them.

ticket: 7864 (new)
tags: pullup
target_version: 1.12.2
Support referrals from Windows Server 2003

Although RFC 6806 Section 7 requires servers to indicate a client
referral in a WRONG_REALM message, Microsoft Windows Server 2003
returns this information in a message with error code
PRINCIPAL_UNKNOWN.  Failure to follow the referral in these messages
prevents referral chasing in Windows Server 2003 forests.  Detect
referral messages of this type by checking for a non-empty
client.realm field in the response, and activate the referral logic in
these cases.

[tlyu@mit.edu: style, comments, and commit message]

ticket: 7856 (new)
target_version: 1.12.2
tags: pullup
Make KDC "status" statements more homogeneous

Generally we want KDC status strings to be concise, informative and follow
some common rules:

- All letters in the status string should be capitalized;
- the words in the status phrase are separated by underscore;
- abbreviations should be avoided.  Some acceptable "standard" acronyms
  are AS_REQ, TGS_REP etc.
- since in almost all cases KDC status is set on error, no need
  to state this fact as part of the status string;
- KDC status string should be an imperative phrase.

For example, "DECRYPT_SERVER_KEY".

This commit is to modify some KDC status messages to follow this format.

Even though KDC status messages are not standardized, it is possible that some
administrators use them in the Kerberos log file processing. Hence, the vast
majority of them are left unchanged pending further investigation (mostly,
feedback from the administrators).
Document 1.12 changes in MIT Kerberos Features doc

ticket: 7857
Use TAILQ macros instead of CIRCLEQ in libdb2

The optimizer in gcc 4.8.1 (but not the current gcc head revision)
breaks the queue.h CIRCLEQ macros, apparently due to an overzealous
strict aliasing deduction.  Use TAILQ macros in the libdb2 mpool code
instead.

ticket: 7860
Use system dictionary for db2 tests again

The built-in word list is not long enough for all of the libdb2 tests
to run properly.  Revert d21a86e47a7cda29225013e08d060095b94b2ee7 and
go back to using the system dictionary if we find one.  However, omit
any lines from the chosen word list which contain non-alphabetical
characters.

ticket: 7860
Simplify iprop update locking and avoid deadlock

Since we are no longer treating the update log like a journal (#7552),
we don't need two-stage update logging.  In kdb5.c, add an update log
entry after each DB change in one step, without getting an explicit
lock.  In kdb_log.c, combine ulog_add_update with ulog_finish_update,
and make ulog_add_update lock the ulog internally.

This change avoids deadlock by removing the only cases where the ulog
is locked before the DB.

ticket: 7861
Factor out ulog serial number status check

Add a new function ulog_get_sno_status, which checks a serial number
and timestamp against the ulog for currency.  Use it in kdb5_util dump
and in ulog_get_entries.  Adjust parse_iprop_header's contract in
dump.c to better match the ulog_get_sno_status contract.

This change causes some minor behavior differences.  kadmind will
check for an empty ulog unless the last serial number matches exactly,
and will never set lastentry when returning UPDATE_FULL_RESYNC_NEEDED
(which was pointless).  kdb5_util dump will recognize a dump file as
current if it exactly matches the last serial number, even if the ulog
is empty; it will be more robust in the presence of non-monotonic
clocks; and it will properly lock around the ulog access.
Lock around more ulog operations

Always lock the ulog when accessing it.  We can currently get away
with some laxness on iprop slaves because they are mostly synchronous,
but hierarchical iprop will allow master and slave operations to take
place concurrently, requiring more strict locking.

Add new functions ulog_get_last and ulog_set_last, which access the
ulog header with locking, and use them in kdb5_util and kpropd.  Add
locking to ulog_replay and ulog_init_header.

ulog_lock and ulog_sync_header are no longer used outside of kdb_log.c
after these changes, so make them static functions and remove the
ulog_ prefix.  Add an unlock_ulog function for clarity.
Simplify ulog_map

Get rid of the caller parameter.  The kproplog semantics (without -R)
for mapping the ulog are simple and almost completely different from
other users of the ulog, so implement them as a static helper in
kproplog.  With hierarchical iprop, kpropd will need the same
semantics as FKCOMMAND and FKADMIND, which were already identical.

Get rid of the db_args parameter, since ulog_map no longer opens the
database after #7552.

Remove an inoperative lseek() call when creating a new ulog file.
Rename ulog_filesize to filesize and compute it from scratch each time
we use it, for easier analysis.  If kdb_hmagic is zero, init the ulog
header but don't skip the rest of the function; it's possible that we
need to expand the ulog file.  Remove an unneeded conditional before
calling extend_file_to for an existing ulog.

ticket: 7855
Maintain complete ulog on iprop slaves

Factor out most of ulog_add_update into a helper function named
store_update, and make ulog_add_update just responsible for assigning
a serial number and timestamp to the update before storing it.  In
ulog_replay, use store_update and ulog_finish_update to add each
update to the ulog in addition to replaying it to the database.  Don't
use incr_ret->lastentry to set kdb_last_sno/kdb_last_time, since it
will have been set properly by adding the individual updates; instead,
just reinitialize the ulog on error.

Slave ulogs use serial numbers provided from upstream, and thus do not
always begin at serial number 1 after a header reset.  As a result, we
must: (A) in store_update, detect the first update (for which we must
assign kdb_first_sno/kdb_first_time) by comparing kdb_num to 0,
instead of by comparing the serial number to 1; (B) in store_update,
detect that we are overwriting the first update by comparing kdb_num
to ulogentries, instead of comparing the serial number to ulogentries;
and (C) in ulog_map, detect that ulogentries changed by verifying the
first and last serial number and timestamp against the actual ulog
entries, rather than simply comparing kdb_last_sno to kdb_num.

Based on code submitted by Richard Basch.

ticket: 7855
Implement kadmind -proponly

The -proponly option causes kadmind to only service the iprop service,
not the kpasswd or kadmin services.  An intermediate slave in a
hierarchical iprop setup runs kadmind -proponly in order to provide
incremental updates to downstream slaves.

Based on code submitted by Richard Basch.

ticket: 7855
Implement kpropd -A

The -A option causes kpropd to contact a specified admin server
(normally an intermediate slave running kadmind -proponly) instead of
the master admin server.

Based on code submitted by Richard Basch.

ticket: 7855
Test iprop slave ulog management

Check the ulog more thoroughly after each operation, including the
principal names we expect in each update entry.  Verify that the slave
ulog contains actual update entries received from master.  Add a
second slave which receives updates from the first.  Test a wider
variety of principal operations.  Add two additional operations after
the full resync to test that incremental updates resume after a full
resync (albeit with some lag).

ticket: 7855
Document hierarchical iprop

Also remove an outdated caveat, but add a new one about policy changes
causing full resyncs.

ticket: 7855