guestagent/test_volume.py leaves a file in /tmp

Since the test mocks() execute and then calls write_to_fstab(), the
file created in /tmp by write_to_fstab() stays around. This is because
write_to_fstab() uses execute to remove the file. Replaced it with an
os.remove() and fixed the test which was counting two calls to
execute().

I'm fixing it because testing for another bug involves running tox
over and over again and leaving these files in /tmp is just a bit
messy.

Change-Id: Ia9e01049687644abd0671a658dffbbdcb4fba4bd
Closes-Bug: #1353064
Merge from oslo-incubator

Merging processutils.py and dependencies from oslo-incubator.
Primarily, I want change d6b55fb but the rest come along for
the ride.

0d7296f Add kwargs to jsonutils.load(s) functions
3310d8d update new requests logger to default WARN
33afb20 Fix broken formatting of processutils.execute log statement
3d90045 Backport code for i18n to check lazy at runtime
433fa0b Make logging_context_format_string optional in log.set_defaults
4d9328c Python 3: enable tests/unit/test_log.py
5fd77eb Ability to customize default_log_levels for each project
6751b30 Remove extra whitespace
726d00a Adjust oslo logging to provide adapter is enabled for
ac92c06 Add default log level for websocket
cb5a804 Move `mask_password` to strutils
d6b55fb Remove `processutils` dependency on `log`
e184dd3 Fix exception message in openstack.common.processutils.execute
ef37e03 Added missing jsonutils.dump() function

Change-Id: I567f514faed5a8467602257d35a98625801daf22
Closes-Bug: #1349890
Mocks utils.execute_with_timeout for mongodb tests

The mongodb test_prepare_from_backup unit test doesn't
mock utils.execute_with_timeout causing it to prompt
for a password and/or hang.

Change-Id: I22d839f9bbd6faf97bba16616e7f2cbcf725ab1e
Closes-Bug: 1353570
Merge "guestagent/test_volume.py leaves a file in /tmp"
recent audit log change broke this LOG.debug message

My recent change to this file as part of the audit logging changes
(b668e85c) introduced this error.

Change-Id: If540d94432835f7f28817b00a39c5c704cb71813
Closes-Bug: #1354107
Merge "Mocks utils.execute_with_timeout for mongodb tests"
guestagent/mysql: Remove unused function

This commit remove the dead function
`replace_mycnf_with_template`.

Closes-bug: 1351361
Change-Id: Ie9a17a972e069b27740c96188f176271599142f4
Correct monkey patching in GuestAgentBackupTest

This change addresses some intermittent failures in tests that we've
been seeing recently.

The failure would typically manifest itself in
test_prepare_from_backup test in test_mongodb_manager.py but in theory
it could occur elsewhere. The good thing about this change is that it
will make the intermittent failures persistent. But, thanks to the fix
for bug 1353570 (change I22d839f) the failures won't happen any
longer.

This change, without the earlier change for bug 1353570 did in fact
serve to make the intermittent failures permanent. The issue is that
GuestAgentBackupTest didn't unpatch execute_with_timeout when it was
done. This meant that tests running after it were running in an
environment where execute_with_timeout was patched. But, just in case
you had a test which didn't itself patch execute_with_timeout and it
happened to run on a thread that hadn't run GuestAgentBackupTest,
you'd get a bad result.

The fix is to include code to save and restore execute_with_timeout in
setUp and tearDown.

Change-Id: I60187145d25524b4e7614d5d8cce9027521db44c
Closes-Bug: #1347337
Merge "recent audit log change broke this LOG.debug message"
Adjusted audit logging for taskmanager module

The taskmanager module audit log messages are unclear, some debug
messages are supposed to be info and vice versa.

Changed logging of various messages to conform to standard.

Change-Id: Idaa46b67096c1fbbacdf730c06c823aaf3e1e255
Partial-Bug: #1324206
Update 'list_users' call to use AGENT_HIGH_TIMEOUT

The list_users call in the guestagent takes about 0.4s - 0.5s per
user, and currently uses the AGENT_LOW_TIMEOUT. This means that it's
possible for the call to easily time out if there are ~11-12 users
on the guest.

This has been causing a lot of false negative failures in the
int-tests, so this updates the 'list_users' call to use the
AGENT_HIGH_TIMEOUT instead of the AGENT_LOW_TIMEOUT.

Closes-bug: 1356085
Change-Id: I589baa0e95febfb63538b79a082cc3dc55f66510
allow both ipv4 and ipv6 hostnames by default

Refactored the code to use netaddr rather than re-implementing IP
address validation. This gives us IPv6 support for free.

DocImpact
The hostname_require_ipv4 configuration option is being
removed in favor of hostname_require_valid_ip. The latter is more
inclusive and validates both IPv4 and IPv6 addresses.

Change-Id: I36c76169f2ab3bb7f54f67db522309a49ae61f7a
Closes-Bug: #1343192
Load trove API extensions using stevedore

The current trove impl loads trove API extensions by searching on a
single file path which is exposed as a conf property. This results in a
less than optimal approach from a consumer extension POV. This change
replaces the single extension path approach with dyanmic loading  using
stevedore. Consumers can now bind into the API extensions using the
'trove.api.extensions' entry point; a standard means to define extension
points. Moreover this change refactors to the
trove.openstack.common.extensions logic into trove.common.extensions.

In addition this change includes base unit tests to ensure the
existing trove proper extension points are loaded and that some basic
checks are in place W/R/T validating trove extension points.

Change-Id: Id3e712b536aef3bc9c9d1603367cdc0d4816b970
Implements: blueprint dynamic-extension-loading
Unit Tests for Mysql replication functionality

Implemented unit tests for the API methods in mysql manager.

Partially implements: blueprint replication-v1

Change-Id: Ifa64630457ddd771d98985823d92b4bab3412890
Merge "Adjusted audit logging for taskmanager module"
Adjusted audit logging for trove.instance module

The trove.instance module is logging some verbose debug data at the
info level, and some info messages at the debug level. Some log
entries are also lacking context because of missing identifiers such
as InstanceID and TenantID.

Changed logging level of various messages to conform to standard,
added references to various ID's as needed.

Change-Id: I3bb569529befa2b2497d2d5a094e3306e569422c
Partial-Bug: #1324206
Mysql guest agent functionality for replication

Implement new strategy for replication
Implement replication strategy for mysql binlog replication.
Supporting methods in mysql service
Implement API methods in mysql manager
Define configuration settings for replication

Co-authored by: Nikhil Manchanda <SlickNik@gmail.com>
Co-authored by: Greg Lucas <glucas@tesora.com>

Partially implements: blueprint replication-v1
Change-Id: I70f0b5c37fe3c2d42426029bb627c141965eb524
show stdout/err from failed command execution

When executing a command the mysql service status command it is useful
to see the actual output of the failing command in debug mode, not just
the exit status.

Add a log_output_on_error parameter to execute_with_timeout() so this
can be reused.

Change-Id: Iff931d0489754303e099e8567bca3237fd476f8f
Closes-Bug: #1340124
Add new checklinks tox environment

This is a new non-voting job that will be used for changes of the
manual. It checks that URLs in the manual are reachable and valid.

Change-Id: I0f34d2fa2a0334276623dfa708f4196a16e20dc1
Move usage_timeout out of guest options

In an effort to further separate the guest and task manager,
the usage_timeout has been moved into the common cfg. The
datastores have no need for this. the default is now set as
the largest timeout, cassandra.

DocImpact: This change modifies a configuration parameter.
partially implements: blueprint refactoring-datastore-options-in-cfg
Co-Author: Michael Basnight <mbasnight@gmail.com>

Change-Id: I3b4b2a9bffb40df4531a78c29d1e155e59436948