etc-update: symlink support for bug #485598

This includes numerous logic adjustments that are needed to support
protected symlinks. The show_diff function now supports arbitrary
file types. For example, a diff between two symlinks looks like this:

-SYM: /foo/bar -> baz
+SYM: /foo/bar -> blah

X-Gentoo-Bug: 485598
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=485598
dispatch-conf: symlink support for bug #485598

This includes numerous logic adjustments that are needed to support
protected symlinks. The new diff_mixed function is used for diffs
between arbitrary file types. For example, a diff between two symlinks
looks like this:

-SYM: /foo/bar -> baz
+SYM: /foo/bar -> blah

X-Gentoo-Bug: 485598
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=485598
CONFIG_PROTECT: protect symlinks, bug #485598

Users may not want some symlinks to get clobbered, so protect them
with CONFIG_PROTECT. Changes were required in the dblink.mergeme method
and the new_protect_filename function.

The changes to dblink.mergeme do 3 things:

 * Move the bulk of config protection logic from dblink.mergeme to a
   new dblink._protect method. The new method only returns 3 variables,
   which makes it easier to understand how config protection interacts
   with the dblink.mergeme code that uses those variables. This is
   important, since dblink.mergeme has so many variables.

 * Initialize more variables at the beginning of dblink.mergeme, since
   those variables are used by the dblink._protect method.

 * Use the variables returned from dblink._protect to trigger
   appropriate behavior later in dblink.mergeme.

The new_protect_filename changes are required since this function
compares the new file to old ._cfg* files that may already exist, in
order to avoid creating duplicate ._cfg* files. In these comparisons,
it needs to handle symlinks differently from regular files.

The unit tests demonstrate operation in many different scenarios,
including:

 * regular file replaces regular file
 * regular file replaces symlink
 * regular file replaces directory
 * symlink replaces symlink
 * symlink replaces regular file
 * symlink replaces directory
 * directory replaces regular file
 * directory replaces symlink

X-Gentoo-Bug: 485598
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=485598
Add egencache --update-pkg-desc-index action.

This adds an egencache --update-pkg-desc-index action which generates
a plain-text index of package names, versions, and descriptions. The
index can then be used to optimize emerge --search / --searchdesc
actions.

X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
Add IndexStreamIterator and MultiIterGroupBy.

This IndexStreamIterator class can be used together with the
pkg_desc_index_line_read function to read an index file incrementally
as a stream.

The MultiIterGroupBy class can be used to iterate over multiple
IndexStreamIterator instances at once, incrementally grouping results
for a particular package from multiple indices, while limiting the
amount of any given index that must be in memory at once.

Both of these classes are used by the  IndexedPortdb class in the next
patch of this series.

X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
Add IndexedPortdb class.

The IndexedPortdb class uses pkg_desc_index to optimize searchs for
package names and descriptions. If the package description index is
missing from a particular repository, then all metadata for that
repository is obtained using the normal pordbapi.aux_get method.

This class only implements a subset of portdbapi functionality that is
useful for searching pkg_desc_index incrementally. For this reason,
the cp_all method returns an ordered iterator instead of a list, so
that search results can be displayed incrementally.

X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
Add IndexedVardb class.

Searching of installed packages is optimized to take advantage of
vardbdbapi._aux_cache, which is backed by vdb_metadata.pickle.
This class only implements a subset of vardbapi functionality that is
useful for searching incrementally. For this reason, the cp_all method
returns an ordered iterator instead of a list, so that search results
can be displayed incrementally.

X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
Add emerge --search-index option.

The new emerge --search-index option, which is enabled by default,
causes pkg_desc_index to be used for search optimization. The search
index needs to be regenerated by egencache after changes are made to
a repository (see the --update-pkg-desc-index action).

For users that would like to modify ebuilds in a repository without
running egencache afterwards, emerge --search-index=n can be used to
get non-indexed search. Alternatively, the user could simply remove
the stale index file, in order to disable the search index for a
particular repository.

In order to conserve memory, indices are read as streams, and
MultiIterGroupBy is used to group results from IndexedPortdb and
IndexedVardb. Stream-oriented search also makes it possible to
display search results incrementally (fixing bug #412471).

X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718