PXE validate() to fail if no Ironic API URL

As part of the deploy process the conductor should pass the URL of the
Ironic API to the deploy ramdisk so that it can talk back to the Ironic
API, but if the URL is not present in the Ironic configuration file or
in the keystone catalog the deploy process will fail in the middle. This
patch adds a verification to the validate() method to make sure the URL
is present before the deploy starts.

Related-Bug: #1290482
Change-Id: I95472660deff0e23a19d0b744f711bbe3acc2191
Do not use __builtin__ in python3

__builtin__ does not exist in Python 3, use
six.moves.builtins instead.

Change-Id: I79b61b5ab67f491ce88015f799ff7d66c9eeb7ee
closes-bug: #1290234
Delete node while waiting for deploy

During a deployment, some drivers set the node.provision_state to DEPLOYWAIT
-- eg, while waiting for a callback from the deploy ramdisk agent. This
change makes it possible to de-provision the node during this window (ie,
clean up the files and network env, and power it off).

Note that this state is explicitly included in the check within the
ConductorManager's do_node_tear_down; it was the API check that prevented
this.

Also modified a user message to indicate that the node is being
provisioned or decommissioned.

Change-Id: Ia67ec635c36f643d99795d91f9dfb10b577f033f
Closes-Bug: #1291056
Stop logging paramiko's DEBUG and INFO messages

When ironic-conductor is run in debug mode with the SSH power driver,
Paramiko logs spurious DEBUG and INFO messages constantly. There is
already sufficient logging within the SSH power driver.

This patch adds paramiko to the list of modules who should only be
logged at or above a WARN level, even in debug mode.

Change-Id: Iccdfebbdc344c33c8fc0ab81f3ab5a176cd43a17
Enable pep8 tests for the Nova Ironic Driver

Enable pep8 tests for the Nova Ironic Driver and fix the current pep8
errors. Being able to run pep8 tests against the driver helps the
development because it captures problems like syntax errors, unbound
variables, duplicated methods, etc...

Change-Id: If1da455bb8669b56267cb3ecf45369d5d83a86b7
Process public API list as regular expressions

In order to allow flexible patterns for public API specification
and to speed up the process of checking whether some request is
performed against the public API, it's reasonable to interpret the
list of public endpoints as regular expressions.

This patch changes the logic of checking whether a request requires
authorisation from searching the endpoint in the set to checking the
endpoint against a list of regular expressions.

Closes-bug: #1251880
Change-Id: I638ca0e20fa7e44fbeeae0d1e4c2f4188fb597a5
Remove redundant nova virt driver test

There were 2 versions of 'test__require_node' that tested the same
thing. Remove one of them.

Change-Id: I3d34b3e23f623839a493e90f12575b39ad4ad434
Add admin_auth_token option for nova driver

The call to nova.context.get_admin_context() never sets an auth_token,
but we should support the idea of having an admin token that's always
known to be good. This adds a new Ironic config option for
'admin_auth_token' and uses it if set.

DocImpact

Closes-bug: 1292676

Change-Id: I73fcf13e2a746be527c234bdc711d95de157a9e5
Merge "Enable pep8 tests for the Nova Ironic Driver"
Remove unused method from timeutils

Ironic no longer uses set_time_override. Therefore there is no need to
invoke clear_time_override anymore.

Change-Id: I502d7bb6bd22b9ad9239cd78c80938d5925347d2
Related-Bug: #1266962
Merge "Add admin_auth_token option for nova driver"
Suppress conductor logging of expected exceptions

RPC methods may return an exception from the ConductorManager as an
expected means to inform the API of an error. Such expected exceptions
should not be logged within the conductor -- otherwise the log will be
overflowing and real (unexpected) exceptions will go unnoticed.

This patch adds a decorator to the RPC methods in ConductorManager to
suppress logging of expected exceptions, while still passing them back
to the API.

Co-Authored-By: David Shrewsbury <shrewsbury.dave@gmail.com>

Change-Id: Ic3107a9af6afeb2a9e772ca0ce0acc45dbc182cb
Closes-bug: #1291509
Merge "Remove unused method from timeutils"
Change deploy validation exception handling

Calls to the deploy validate() method may throw an InvalidParameterValue
exception during node deploy or node tear down. It's more logical to catch
this and throw an InstanceDeployFailure exception instead.

This addresses comments raised during review of change ID
Ic3107a9af6afeb2a9e772ca0ce0acc45dbc182cb.

The test_do_node_deploy_validate_fail and test_do_node_tear_down_validate_fail
tests intentionally still have their mocked object raise the
InvalidParameterValue exception to test that it gets transformed into
an InstanceDeployFailure exception.

Change-Id: I754884d1a012a9e67d2cc7a74e3121e0e02e76ec
Merge "Delete node while waiting for deploy"
Merge "Suppress conductor logging of expected exceptions"
Merge "Stop logging paramiko's DEBUG and INFO messages"
Conductor de-registers on shutdown

Conductor service should de-register itself upon shutdown.
This will cause RPC requests to immediately stop routing to it
rather than there being an unnecessary period of malfunction
before the next heartbeat cycle.

A new unit test (test_stop_unregisters_conductor) is added to
validate the de-registration of the conductor. Test hostname
values are coalesced into a single variable for consistency.

Co-Authored-By: David Shrewsbury <shrewsbury.dave@gmail.com>

Change-Id: I1feea2c0165cde36f2abb95c3fa3484c73dfd81c
Make sure auth_url exists and is not versionless

When getting the Ironic API service URL from the Keystone catalog make
sure that the Keystone API endpoint is present and that it contains the
right version of the API.

Closes-Bug: #1290482
Change-Id: I80fc74a1aea02074baba4839df83e0e694bc8ea6
Better instance-not-found handling within IronicDriver

Improve the handling of failed spawns and multiple deletes within the
IronicDriver.

Change-Id: I284f4669ba14db7bb33b5a53512b6a3822ad170c