Refactor bashrc scripts sourcing

Creates two new helper functions __try_source and __source_env_files to simplify
__source_all_bashrcs.
Add profile-formats to profile_complex
package.bashrc: per profile, per-package bashrc mechanism

Profiles can define per-package bashrc files to be sourced before emerging.
Each line in package.bashrc must be an atom name then a list of space-delimited
bashrc files (stored in $profile/bashrc/).
bin/ebuild: allow useful phases with pkg_config

It makes sense to run pkg_config by itself, but special phases like
"clean", "digest", and "manifest" can be useful. For example, if the
developer modifies the ebuild, then it's useful to regenerate the
manifest and use "clean" to discard the stale environment.

Fixes: bfa98d7a5b7a ("fixed ebuild so")
Reported-by: Vlastimil Babka <caster@gentoo.org>
Reviewed-by: Alexander Berntsen <bernalex@gentoo.org>
depgraph: fix bug #526160

This fixes _dep_check_composite_db to mask packages that aren't the
highest visible match, but only if an update is desirable. This causes
desirable updates to get pulled in for cases like bug #526160. The
included unit test simulates the virtual/pypy update that triggered
the bug.

X-Gentoo-Bug: 526160
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=526160
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
CONFIG_PROTECT: handle non-existent files

This fixes the ConfigProtect class, etc-update, and dispatch-conf to
account for non-existent files (rather than directories) that are
listed directly in CONFIG_PROTECT. It has been valid to list files
directly in CONFIG_PROTECT since bug #14321. However, the support for
non-existent files added for bug #523684 did not include support for
non-existent files listed directly in CONFIG_PROTECT.

Fixes: 5f7b4865ecaf ("dblink.mergeme: implement bug #523684")
X-Gentoo-Bug: 523684
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=523684
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
bin/bashrc-functions.sh: remove portageq function

Since commit 0cc4c1ac21a2ea94cfb1f6ff4b461a9e349d47df,
$PORTAGE_BIN_PATH/portageq no longer exists, which breaks the portageq
shell function inside bashrc-functions.sh. Note that
$PORTAGE_BIN_PATH/ebuild-helpers/portageq is in $PATH, so we can simply
remove the redundant function.

Fixes: 0cc4c1a ("Install Portage using setup.py")
X-Gentoo-Bug: 524964
X-Gentoo-URL: https://bugs.gentoo.org/show_bug.cgi?id=524964
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
Introduce eqalog and eqawarnlog functions.

These functions are to be used for creating a log of QA violations in a
machine-readable format.

Stored in ${T]/qa.log, the log consists of one entry per line in the
following format: violation-tag data

For example:
deprecated-directory /usr/man
deprecated-directory /usr/info
world-writable /var/db/foo/bar

Signed-off-by: Zac Medico <zmedico@gentoo.org>
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
install-qa-check.d/05double-D: Write to log and improve consistency.

Present the list of offending files newline-delimitered for consistency
with other checks.

Signed-off-by: Zac Medico <zmedico@gentoo.org>
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
install-qa-check.d/90world-writable: Write log and general cleanup.

Use eqawarn instead of __vecho for visibility.

Present the list of offending files newline-delimitered for consistency
with other checks.

Signed-off-by: Zac Medico <zmedico@gentoo.org>
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
dep_zapdeps: handle circular deps with --onlydeps

This fixes a case with --onlydeps were dep_zapdeps would pull in an
avoidable direct circular dependency on an onlydeps node. The logic
changes only apply to --onlydeps, so there's no chance of regressions
for cases when --onlydeps is not enabled.

X-Gentoo-Bug: 524916
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=524916
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
Add btrfs.* to default PORTAGE_XATTR_EXCLUDE (527636)

In bug #527636 a user reported that install-xattr failed to copy
btrfs.compression attributes when doins was installing a file from
$FILESDIR. It is unlinkly that a package will have a valid use for
btrfs.* attributes when installing files to $D. Therefore, exclude
btrfs.* attributes in order to avoid reports like this in the
future.

X-Gentoo-Bug: 527636
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=527636
Acked-by: Brian Dolbec <dolsen@gentoo.org>
_dep_check_composite_db: fix bug #526160

The fixes various forms of buggy behavior involving virtuals handling
and EAPI 5 subslots. The code at fault is part of the support for
bug #141118 (virtuals lookahead). This code was written long before
subslots existed, and because of them it needs to be updated now.

Previously, the relevant virtuals code would only select one virtual
from each slot for lookahead. Now, it needs to select one from each
slot/subslot pair. The buggy forms of behavior fixed by this change
can be difficult to reproduce, since they only express themselves
under some rare circumstances.

In addition to the _dep_check_composite_db fixes, there is also a
related fix inside _expand_new_virtuals, which is required so that
_slot_operator_update_probe can reliably check whether or not a
particular input atom was selected. Without this change, the
_dep_check_composite_db changes will cause the test from commit
d3be49fe6827aa1974856dffe6d5a1aca80a7bed to fail in a way that is
very similar to the failure reported in bug #526160.

Fixes: d3be49fe6827 ("depgraph: fix bug #526160")
X-Gentoo-Bug: 526160
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=526160
Acked-by: Brian Dolbec <dolsen@gentoo.org>
LocationsManager: enable user package.bashrc

In /etc/portage/profile, we already have support for using directories
in the place of regular config files (portage1_directories = True). So,
go ahead and enable package.bashrc there too.

We could also support /etc/portage/package.bashrc, but that would be
more complicated since /etc/portage/bashrc is the long-standing location
of the user's bashrc which applies to all packages, and package.bashrc
would instead expect /etc/portage/bashrc to be a directory containing
per-package bashrc files. So for now, we only support package.bashrc
inside /etc/portage/profile.

Note that since /etc/portage/profile is not part of a repository,
there's no associated layout.conf. So, there's no way to enable the
profile-env profile-formats setting in /etc/portage/profile, except by
hardcoding.

Signed-off-by: Zac Medico <zmedico@gentoo.org>
Acked-by: Brian Dolbec <dolsen@gentoo.org>
If a binhost file sets a TTL header, honor it.

Reviewed-on: https://chromium-review.googlesource.com/225279
Reviewed-on: https://chromium-review.googlesource.com/225423
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Zac Medico <zmedico@gentoo.org>
Acked-by: Brian Dolbec <dolsen@gentoo.org>
etc-update: symlink support for bug #485598

This includes numerous logic adjustments that are needed to support
protected symlinks. The show_diff function now supports arbitrary
file types. For example, a diff between two symlinks looks like this:

-SYM: /foo/bar -> baz
+SYM: /foo/bar -> blah

X-Gentoo-Bug: 485598
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=485598
dispatch-conf: symlink support for bug #485598

This includes numerous logic adjustments that are needed to support
protected symlinks. The new diff_mixed function is used for diffs
between arbitrary file types. For example, a diff between two symlinks
looks like this:

-SYM: /foo/bar -> baz
+SYM: /foo/bar -> blah

X-Gentoo-Bug: 485598
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=485598
CONFIG_PROTECT: protect symlinks, bug #485598

Users may not want some symlinks to get clobbered, so protect them
with CONFIG_PROTECT. Changes were required in the dblink.mergeme method
and the new_protect_filename function.

The changes to dblink.mergeme do 3 things:

 * Move the bulk of config protection logic from dblink.mergeme to a
   new dblink._protect method. The new method only returns 3 variables,
   which makes it easier to understand how config protection interacts
   with the dblink.mergeme code that uses those variables. This is
   important, since dblink.mergeme has so many variables.

 * Initialize more variables at the beginning of dblink.mergeme, since
   those variables are used by the dblink._protect method.

 * Use the variables returned from dblink._protect to trigger
   appropriate behavior later in dblink.mergeme.

The new_protect_filename changes are required since this function
compares the new file to old ._cfg* files that may already exist, in
order to avoid creating duplicate ._cfg* files. In these comparisons,
it needs to handle symlinks differently from regular files.

The unit tests demonstrate operation in many different scenarios,
including:

 * regular file replaces regular file
 * regular file replaces symlink
 * regular file replaces directory
 * symlink replaces symlink
 * symlink replaces regular file
 * symlink replaces directory
 * directory replaces regular file
 * directory replaces symlink

X-Gentoo-Bug: 485598
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=485598
Remove redundant PORTAGE_XATTR_EXCLUDE defaults (527636)

In install.py and movefile.py there were some redundant default
PORTAGE_XATTR_EXCLUDE settings. The default settings in make.globals
should be sufficient. Therefore, remove the redundant settings so that
we don't have to maintain them.

Fixes: 2fcdb5f36fac ("Add btrfs.* to default PORTAGE_XATTR_EXCLUDE (527636)")
X-Gentoo-Bug: 527636
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=527636
Acked-by: Brian Dolbec <dolsen@gentoo.org>
Acked-by: Anthony G. Basile <blueness@gentoo.org>
doebuild: fix bug #528272

The doebuild function has a boolean "unpacked" variable which it sets
to True if it determines that the "unpack" phase has already executed
and the last modification times of the distfiles are older than the
last modification time of WORKDIR. The code which sets the "unpacked"
flag does not need to run unless the current phase is supposed to have
a build directory. Therefore, disable this code for all phases that are
not supposed to have a build directory. This fixes incorrect behavior
of the "fetch" phase as reported in bug #528272.

X-Gentoo-Bug: 528272
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=528272
Acked-by: Alexander Berntsen <bernalex@gentoo.org>