Add btrfs.* to default PORTAGE_XATTR_EXCLUDE (527636)

In bug #527636 a user reported that install-xattr failed to copy
btrfs.compression attributes when doins was installing a file from
$FILESDIR. It is unlinkly that a package will have a valid use for
btrfs.* attributes when installing files to $D. Therefore, exclude
btrfs.* attributes in order to avoid reports like this in the
future.

X-Gentoo-Bug: 527636
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=527636
Acked-by: Brian Dolbec <dolsen@gentoo.org>
_dep_check_composite_db: fix bug #526160

The fixes various forms of buggy behavior involving virtuals handling
and EAPI 5 subslots. The code at fault is part of the support for
bug #141118 (virtuals lookahead). This code was written long before
subslots existed, and because of them it needs to be updated now.

Previously, the relevant virtuals code would only select one virtual
from each slot for lookahead. Now, it needs to select one from each
slot/subslot pair. The buggy forms of behavior fixed by this change
can be difficult to reproduce, since they only express themselves
under some rare circumstances.

In addition to the _dep_check_composite_db fixes, there is also a
related fix inside _expand_new_virtuals, which is required so that
_slot_operator_update_probe can reliably check whether or not a
particular input atom was selected. Without this change, the
_dep_check_composite_db changes will cause the test from commit
d3be49fe6827aa1974856dffe6d5a1aca80a7bed to fail in a way that is
very similar to the failure reported in bug #526160.

Fixes: d3be49fe6827 ("depgraph: fix bug #526160")
X-Gentoo-Bug: 526160
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=526160
Acked-by: Brian Dolbec <dolsen@gentoo.org>
LocationsManager: enable user package.bashrc

In /etc/portage/profile, we already have support for using directories
in the place of regular config files (portage1_directories = True). So,
go ahead and enable package.bashrc there too.

We could also support /etc/portage/package.bashrc, but that would be
more complicated since /etc/portage/bashrc is the long-standing location
of the user's bashrc which applies to all packages, and package.bashrc
would instead expect /etc/portage/bashrc to be a directory containing
per-package bashrc files. So for now, we only support package.bashrc
inside /etc/portage/profile.

Note that since /etc/portage/profile is not part of a repository,
there's no associated layout.conf. So, there's no way to enable the
profile-env profile-formats setting in /etc/portage/profile, except by
hardcoding.

Signed-off-by: Zac Medico <zmedico@gentoo.org>
Acked-by: Brian Dolbec <dolsen@gentoo.org>
If a binhost file sets a TTL header, honor it.

Reviewed-on: https://chromium-review.googlesource.com/225279
Reviewed-on: https://chromium-review.googlesource.com/225423
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Zac Medico <zmedico@gentoo.org>
Acked-by: Brian Dolbec <dolsen@gentoo.org>
etc-update: symlink support for bug #485598

This includes numerous logic adjustments that are needed to support
protected symlinks. The show_diff function now supports arbitrary
file types. For example, a diff between two symlinks looks like this:

-SYM: /foo/bar -> baz
+SYM: /foo/bar -> blah

X-Gentoo-Bug: 485598
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=485598
dispatch-conf: symlink support for bug #485598

This includes numerous logic adjustments that are needed to support
protected symlinks. The new diff_mixed function is used for diffs
between arbitrary file types. For example, a diff between two symlinks
looks like this:

-SYM: /foo/bar -> baz
+SYM: /foo/bar -> blah

X-Gentoo-Bug: 485598
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=485598
CONFIG_PROTECT: protect symlinks, bug #485598

Users may not want some symlinks to get clobbered, so protect them
with CONFIG_PROTECT. Changes were required in the dblink.mergeme method
and the new_protect_filename function.

The changes to dblink.mergeme do 3 things:

 * Move the bulk of config protection logic from dblink.mergeme to a
   new dblink._protect method. The new method only returns 3 variables,
   which makes it easier to understand how config protection interacts
   with the dblink.mergeme code that uses those variables. This is
   important, since dblink.mergeme has so many variables.

 * Initialize more variables at the beginning of dblink.mergeme, since
   those variables are used by the dblink._protect method.

 * Use the variables returned from dblink._protect to trigger
   appropriate behavior later in dblink.mergeme.

The new_protect_filename changes are required since this function
compares the new file to old ._cfg* files that may already exist, in
order to avoid creating duplicate ._cfg* files. In these comparisons,
it needs to handle symlinks differently from regular files.

The unit tests demonstrate operation in many different scenarios,
including:

 * regular file replaces regular file
 * regular file replaces symlink
 * regular file replaces directory
 * symlink replaces symlink
 * symlink replaces regular file
 * symlink replaces directory
 * directory replaces regular file
 * directory replaces symlink

X-Gentoo-Bug: 485598
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=485598
Remove redundant PORTAGE_XATTR_EXCLUDE defaults (527636)

In install.py and movefile.py there were some redundant default
PORTAGE_XATTR_EXCLUDE settings. The default settings in make.globals
should be sufficient. Therefore, remove the redundant settings so that
we don't have to maintain them.

Fixes: 2fcdb5f36fac ("Add btrfs.* to default PORTAGE_XATTR_EXCLUDE (527636)")
X-Gentoo-Bug: 527636
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=527636
Acked-by: Brian Dolbec <dolsen@gentoo.org>
Acked-by: Anthony G. Basile <blueness@gentoo.org>
doebuild: fix bug #528272

The doebuild function has a boolean "unpacked" variable which it sets
to True if it determines that the "unpack" phase has already executed
and the last modification times of the distfiles are older than the
last modification time of WORKDIR. The code which sets the "unpacked"
flag does not need to run unless the current phase is supposed to have
a build directory. Therefore, disable this code for all phases that are
not supposed to have a build directory. This fixes incorrect behavior
of the "fetch" phase as reported in bug #528272.

X-Gentoo-Bug: 528272
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=528272
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
Add support for SUSE based distros in etc-update

Add support for reading/merging *.rpmnew files created in SUSE based distros.
Configuration from etc-update will be read:
- from portage config vars in Gentoo based distros
- from portage config vars in non Gentoo distros if portage is installed
- from /etc/sysconfig/etc-update in SUSE distros if portage is absent

Signed-off-by: Michal Hrušecký <miska@gentoo.org>
X-Gentoo-Bug: 456128
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=456128
Acked-by: Brian Dolbec <dolsen@gentoo.org>
Acked-by: Zac Medico <zmedico@gentoo.org>
man/ebuild.5: document assert for bug #528760

X-Gentoo-Bug: 528760
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=528760
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
_selinux.setexec: improve failure message (525726)

This converts "OSError: [Errno 22] Invalid argument" into a more
meaningful error message that is designed to guide users in the
right direction. Also, add an import that was missing for the "sys"
module, since it is referenced in the error paths of many functions
in this module.

X-Gentoo-Bug: 525726
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=525726
Acked-by: Brian Dolbec <dolsen@gentoo.org>
Acked-by: Zac Medico <zmedico@gentoo.org>
portageq: fix eroot parameter (bug #529200)

The portageq eroot parameter has been broken since commit
c9f6aa9f0151adb3c86706eaef1914cdbdcf2b6d, due to premature instantiation
of portage.settings (before the ROOT variable was set). Premature access
to the portage.settings attribute must be avoided by using other
available means to determine the EPREFIX.

Fixes: c9f6aa9f0151 ("Add cross-prefix support")
X-Gentoo-Bug: 529200
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=529200
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
fs_template._ensure_dirs: handle EEXIST (529120)

There was a race inside fs_template._ensure_dirs which could cause it to
raise EEXIST if a concurrent process created the directory after
os.path.exists returned False. Fix it by using the util.ensure_dirs
function, which already handles EEXIST.

X-Gentoo-Bug: 529120
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=529120
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
unprivileged mode: generate PORTAGE_DEPCACHEDIR

For unprivileged mode, if PORTAGE_DEPCACHEDIR is unset and the default
PORTAGE_DEPCACHEDIR setting does not refer to a writable directory
(or there are not sufficient permissions to create it), then
automatically make PORTAGE_DEPCACHEDIR relative to the current target
root (which should always be writable for unprivileged mode). Also, in
create_trees, propagate the automatically generated PORTAGE_DEPCACHEDIR
setting to the config instance that is instantiated for ROOT = "/".

The result is that unprivileged mode will get a writable
PORTAGE_DEPCACHEDIR by default, and the default can be overridden by
setting the PORTAGE_DEPCACHEDIR variable.

Fixes: 1364fcd89384 ("Support unprivileged mode for bug #433453.")
X-Gentoo-Bug: 433453
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=433453
Acked-by: Alexander Berntsen <bernalex@gentoo.org>
_slot_operator_update_probe: fix bug #528610

This fixes a case inside _slot_operator_update_probe where it would
select an inappropriate replacement_parent of a lower version than
desired. The problem is solved by rejecting replacement_parent if its
version is lower than the existing parent, and a downgrade is not
desired.

X-Gentoo-Bug: 528610
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=528610
Acked-by: Brian Dolbec <dolsen@gentoo.org>
unprivileged mode: use first_existing helper func

Split out a first_existing function, in order to improve logic related
to the _unprivileged_mode function so that it checks whether it's
possible to create the specified target root (instead of requiring that
the target root already exists).

Acked-by: Alexander Berntsen <bernalex@gentoo.org>
unprivileged mode: fix cross-prefix support

In commit 1364fcd89384c9f60e6d72d7057dc00d8caba175, EROOT calculation
in portage.data did not account for cross-prefix support. This is
fixed by using new _target_root and _target_eprefix functions to
perform the calculation. The _target_eprefix function is also useful
in portageq, where the target EPREFIX needs to be known before
portage.settings is instantiated.

Fixes 1364fcd89384 ("Support unprivileged mode for bug #433453.")
Acked-by: Brian Dolbec <dolsen@gentoo.org>
NewsManager.getUnreadItems: handle EROFS (490732)

When getUnreadItems tries to lock the news.unread file, it's safe to
ignore EROFS. This is handled with a ReadOnlyFileSystem exception
raised from the portage.locks.lockfile function.

X-Gentoo-Bug: 490732
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=490732
Acked-by: Brian Dolbec <dolsen@gentoo.org>
dblink: case insensitive support for bug #524236

This adds a dblink._contents attribute with methods that provide
an interface for contents operations with "implicit" case handling.
The new methods are implemented in a separate
ContentsCaseSensitivityManager class, in order to avoid adding more
bloat to vartree.py.

X-Gentoo-Bug: 524236
X-Gentoo-Url: https://bugs.gentoo.org/show_bug.cgi?id=524236
Acked-by: Brian Dolbec <dolsen@gentoo.org>