Add btrfs.* to default PORTAGE_XATTR_EXCLUDE (527636)

In bug #527636 a user reported that install-xattr failed to copy
btrfs.compression attributes when doins was installing a file from
$FILESDIR. It is unlinkly that a package will have a valid use for
btrfs.* attributes when installing files to $D. Therefore, exclude
btrfs.* attributes in order to avoid reports like this in the
future.

X-Gentoo-Bug: 527636
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=527636
Acked-by: Brian Dolbec <dolsen@gentoo.org>
_dep_check_composite_db: fix bug #526160

The fixes various forms of buggy behavior involving virtuals handling
and EAPI 5 subslots. The code at fault is part of the support for
bug #141118 (virtuals lookahead). This code was written long before
subslots existed, and because of them it needs to be updated now.

Previously, the relevant virtuals code would only select one virtual
from each slot for lookahead. Now, it needs to select one from each
slot/subslot pair. The buggy forms of behavior fixed by this change
can be difficult to reproduce, since they only express themselves
under some rare circumstances.

In addition to the _dep_check_composite_db fixes, there is also a
related fix inside _expand_new_virtuals, which is required so that
_slot_operator_update_probe can reliably check whether or not a
particular input atom was selected. Without this change, the
_dep_check_composite_db changes will cause the test from commit
d3be49fe6827aa1974856dffe6d5a1aca80a7bed to fail in a way that is
very similar to the failure reported in bug #526160.

Fixes: d3be49fe6827 ("depgraph: fix bug #526160")
X-Gentoo-Bug: 526160
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=526160
Acked-by: Brian Dolbec <dolsen@gentoo.org>
LocationsManager: enable user package.bashrc

In /etc/portage/profile, we already have support for using directories
in the place of regular config files (portage1_directories = True). So,
go ahead and enable package.bashrc there too.

We could also support /etc/portage/package.bashrc, but that would be
more complicated since /etc/portage/bashrc is the long-standing location
of the user's bashrc which applies to all packages, and package.bashrc
would instead expect /etc/portage/bashrc to be a directory containing
per-package bashrc files. So for now, we only support package.bashrc
inside /etc/portage/profile.

Note that since /etc/portage/profile is not part of a repository,
there's no associated layout.conf. So, there's no way to enable the
profile-env profile-formats setting in /etc/portage/profile, except by
hardcoding.

Signed-off-by: Zac Medico <zmedico@gentoo.org>
Acked-by: Brian Dolbec <dolsen@gentoo.org>
If a binhost file sets a TTL header, honor it.

Reviewed-on: https://chromium-review.googlesource.com/225279
Reviewed-on: https://chromium-review.googlesource.com/225423
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Zac Medico <zmedico@gentoo.org>
Acked-by: Brian Dolbec <dolsen@gentoo.org>
Add egencache --update-pkg-desc-index action.

This adds an egencache --update-pkg-desc-index action which generates
a plain-text index of package names, versions, and descriptions. The
index can then be used to optimize emerge --search / --searchdesc
actions.

X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
Add IndexStreamIterator and MultiIterGroupBy.

This IndexStreamIterator class can be used together with the
pkg_desc_index_line_read function to read an index file incrementally
as a stream.

The MultiIterGroupBy class can be used to iterate over multiple
IndexStreamIterator instances at once, incrementally grouping results
for a particular package from multiple indices, while limiting the
amount of any given index that must be in memory at once.

Both of these classes are used by the  IndexedPortdb class in the next
patch of this series.

X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
Add IndexedPortdb class.

The IndexedPortdb class uses pkg_desc_index to optimize searchs for
package names and descriptions. If the package description index is
missing from a particular repository, then all metadata for that
repository is obtained using the normal pordbapi.aux_get method.

This class only implements a subset of portdbapi functionality that is
useful for searching pkg_desc_index incrementally. For this reason,
the cp_all method returns an ordered iterator instead of a list, so
that search results can be displayed incrementally.

X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
Add IndexedVardb class.

Searching of installed packages is optimized to take advantage of
vardbdbapi._aux_cache, which is backed by vdb_metadata.pickle.
This class only implements a subset of vardbapi functionality that is
useful for searching incrementally. For this reason, the cp_all method
returns an ordered iterator instead of a list, so that search results
can be displayed incrementally.

X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718
Add emerge --search-index option.

The new emerge --search-index option, which is enabled by default,
causes pkg_desc_index to be used for search optimization. The search
index needs to be regenerated by egencache after changes are made to
a repository (see the --update-pkg-desc-index action).

For users that would like to modify ebuilds in a repository without
running egencache afterwards, emerge --search-index=n can be used to
get non-indexed search. Alternatively, the user could simply remove
the stale index file, in order to disable the search index for a
particular repository.

In order to conserve memory, indices are read as streams, and
MultiIterGroupBy is used to group results from IndexedPortdb and
IndexedVardb. Stream-oriented search also makes it possible to
display search results incrementally (fixing bug #412471).

X-Gentoo-Bug: 525718
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525718