Implement instance rebuild in nova.virt.driver

This implementation of the rebuild operation allows the use of
the --preserve-ephemeral option. The default implementation in
Nova does not support this option.

Note: Image validation is not handled with this change.

Change-Id: I933658d139dc16ef1a65378b0084b4baf6180651
Closes-Bug: #1307754
Support serial console access

This patch implements a console driver IPMIShellinaboxConsole that
uses ipmitool + shellinabox. The console can be accessed from a web
browser with the url returned from the driver. The console_utils
module is created for the common console functions.

The following new config options under the 'console' group are
introduced: terminal, terminal_cert_dir, terminal_pid_dir,
subprocess_checking_interval and subprocess_timeout.

When invoking the shellinaboxd console process, subprocess.Popen
is used instead of openstack.common.processutils because the latter
hangs for long running daemon process.

Co-Authored-By: Ling Gao <linggao@us.ibm.com>
Co-Authored-By: Matt Wagner <matt.wagner@redhat.com>

blueprint serial-console-access

Change-Id: I89a21e4d6969b91386896d1abc7035f9ba0232d9
Add create() and destroy() to Node

What it says. Also make the code use them vs direct calls to dbapi. This
also removes passing context to node.refresh() and node.save() since
context is cached on the Node object on create.

Partial-Bug: 1314732

Change-Id: Id3fafe1eae70954095ed94dec86e4583b23c5af8
Fix ironic node state comparison

Using "is not" is an identity comparison and doesn't compare actual
values. If NOSTATE were to be changed from None to a string, this
would likely break.

Change-Id: I96706e0a3a39971c3661210dbd13a8bc990a8933
Closes-Bug: #1315050
Updated from global requirements

Change-Id: Ia4aec9704d5ad6adc748a7373fb1d72bbbf64f85
Implement more robust caching for master images

This patch introduces caching for master images, which
are no longer used. This patch also introduces clean up procedure,
which is triggered every time cache size increases or link
to master images is deleted. Algorithm is following:
- Images are eligible for deletion, if their link count equals 1.
  Images that has link count >1 are not considered for deletion,
  because it anyway won't reclaim any disk space.
- All images older than configurable TTL are deleted
- Then the oldest images are deleted, until size of cache fits
  into configurable value or no more images eligable for deletion.

As this change required rethinking how we cache images, I also
did a refactor and grouped related functions under ImageCache class.

Change-Id: I1c849fe209c4dbb15e075dbb34c4ea33010a8883
Closes-Bug: #1199665
Merge "Implement instance rebuild in nova.virt.driver"
Merge "Add create() and destroy() to Node"
Merge "Implement more robust caching for master images"
Merge "Fix ironic node state comparison"
Merge "Support serial console access"
Factoring out PXE and TFTP functions

Multiple drivers need to support neutron and Ironic's PXE and TFTP
server. Factoring them out into a common file will make that
transition possible. I've moved all the generic calls into neutron
and reconfigured the PXE driver to work with the new functions.

Change-Id: I181ffabe673636b628329592e623e6425da54d6d
(cherry picked from commit c51d4567de2721bf64612cc0eaff02a7360e9061)
Adding a reference driver for the agent

This driver handles communication with the Ironic Python Agent's REST
API and controls the agent through the deploy and tear down process.
It utilizes the nodeless vendor driver passthru to communicate with
newly booted agents. Things not included yet are support for
Neutron, agent registration/discovery, and writing an image to the
drive before deploy is called.

The original source for this driver was on GitHub. For history,
check: https://github.com/rackerlabs/ironic-teeth-driver

Depends on:
https://review.openstack.org/#/c/81391/
https://review.openstack.org/#/c/81919/

Implements: blueprint agent-driver
Change-Id: Ic6ad8703304af3041c48e9ade32eccbffa78ec9c
(cherry picked from commit fc784a10c543a87360297f524af1172ba1e1454a)
Drivers determine acceptable power states

The list of acceptable power states for a Node to be in when Nova is
calling deploy, update, and destroy were set globally.
This patch allows the deploy drivers to determine
which states are acceptable for each. I added a new validate
method to the DeployInterface class that can be overriden if
a driver determines it needs different settings (such as the agent
driver). The current function provides sane defaults and the
same behavior as before for any driver not overriding the function.

Change-Id: Iefec4396c323d9746bb0ee84b015765a4adbb476
Closes-Bug: 1306202
(cherry picked from commit 8ae97a57d6211fe8f4429f374e22690d2be722a6)
Adding swift temp url support

This patch will allow properly configured Glance servers to return a temporary
URL for an object hosted on Swift. It will require Glance to use Swift
as its backend. A temporary URL will let the agent download an image from
Glance without requiring an auth_token, which gives access more than just
Glance. The easiest way to use it is to enable direct_url in Glance, but
you can set the appropriate config options to avoid needing to enable it.

We/I need to add a note in the docs about Swift being a possible dependency
for Ironic deploys using IPA, along with how to set the Temp URL key.

Swift perfomance concerns will be addressed in this blueprint:
https://blueprints.launchpad.net/ironic/+spec/improve-swift-agent-downloads

SecurityImpact
DocImpact

Implements: blueprint swift-temp-urls
Change-Id: I10e4784eee63e8edc9ba30a9c5004a08aa3a6d8e
(cherry picked from commit 7fb05fa546c2c51e70d50a54dfc0e1cfe9275d92)