Move rootwrap code to openstack.common

Copies current nova-rootwrap code to openstack.common, so that it
can be reused by Cinder and Quantum.

Implements blueprint common-rootwrap.

Before it can be used in projects, update.py needs to grow the
capability to deploy files in bin/ and etc/, as well as replacing
a placeholder text by the destination project name in source files
and binary names.

In this proposed version, the placeholder text is "oslo".

Change-Id: I8655d5b3cccacd1cc2225aa539339fb478615422
Allow update.py copy config files and binaries

Implements blueprint common-binaries

Add support to update.py for copying config files and binaries matching

  etc/oslo/$module*.conf
  bin/oslo-$module*

into the destination project after replacing 'oslo' with the destination
project name.

This is specifically intended for rootwrap but may be applicable in
other cases in the future.

Change-Id: If8a3a815ef452b0dd73ed28cb61a7ff16a9dee5c
fix intermittent failures with deferred version test

Need to get rid of the deferred string approach as
the problem was overriding string methods is hard and
flaky.  There is just no other documented
argparse hook to allow deferring of the version
calculation as version passed in has to be a str
or buffer. Added a test to try longer string and even
run it in a loop just to be sure.

Fixes LP #1087575

Change-Id: Icf83944d67b8ce5d3230627f3de10501da3276f2
Fix debug message in multicall of rpc.amqp

Change-Id: Ib5ed65df937e9fe4e36cdf7d0ce92b627f22dbd3
Close kombu.connection.BrokerConnection correctly

Make kombu.connection.BrokerConnection use release() to close the connection.
It has no such attribute as close(), since AttributeError is in
connection_errors, the fault was previous concealed by the following catch
statement.

Change-Id: I9f7da2c64f9d408aa2a9ddf723ea83291887cff6
Don't use exclusive queues for fanout and direct

The use of exclusive queues can cause issues on reconnect if the
rabbit server hasn't deleted the queue already. We might as well
use exclusive=False so that we can reconnect to the queue if
necessary.

Fixes bug 1094358

Change-Id: I35bd903737d6054e8856b0003b0b8af97fd21c91
Don't rely on os.wait() blocking

Before this fix which was included in eventlet 0.10.0:

https://bitbucket.org/which_linden/eventlet/issue/92/eventletgreen-override-of-oswaitpid

the os.wait() wrapper in eventlet would not block if no child processes
had exited. This was clearly buggy behaviour and os.wait() no blocks
correctly. However, it appears it now does not return if the parent process
was interrupted by a signal. This means that you can't kill the parent process
and have the child processes cleaned up gracefully.

To avoid all these issues, switch to non-blocking wait behaviour. We use
WNOHANG with waitpid() to poll for an exited child process and, if there
is none, we yield to other green threads and sleep for a short period.
This means excessive CPU usage, though, which will be tracked by
bug #1095346.

Check pid to avoid some unnecessary warning/info messages in the logs

Fixes LP #1094076

Change-Id: I783fac652376c2daa9d591403b9f4c1fe9523043
Add a rpc envelope format version number.

This patch adds a message envelope that includes a envelope format
version number.  This message envelope will allow us to embed
additional metadata later on, such as a signature for the message
payload.

Up to this point, we've deferred message serialization as a
responsibility of the messaging library we're using by passing it a
message as Python types and letting it deal with how to pass it over a
network.  This patch adds json serialization in the rpc layer of the
application message payload before passing the message down into the
messaging library.

There are some benefits to be gained by doing a pass at serialization
ourselves.  As an example, we occasionally hit serialization bugs that
only affect some messaging drivers.  The kombu driver has always had a
nice advantage.  It uses anyjson internally for serializing messages,
which we hook into using our jsonutils module.  When there is a problem
serializing, we automatically use to_primitive() to fix it.  This patch
allows all drivers to take advantage of this automatic message fix-up.

This also creates a convenient common hook point for messages coming in
and out of the system, regardless of the driver in use.

While this changes the base format of the messages sent between nodes,
it has been done in a backwards compatible manner.  The new message
format will not be used by default.  The idea is that all nodes will be
upgraded to a version that is capable of receiving the new format
(Grizzly) before switching it on.  We will turn it on post-Grizzly.

Implement blueprint version-rpc-messages.

Change-Id: Ib6b2d11ca42abaa64c40986d72233e7048e504a0
Fix test cases in tests.unit.test_service

Be patient in test_service while waiting service to be spawned or killed,
because:
    * Service spawning has a limit of one process a second.
    * A test server in the cloud might have limited resources, the time there
      might be several times slower.

Change-Id: Idab1d1158ecc7564d0b1f9cdd32496496ff5b3f8
Ensure proxy callback threads exit during tests

On the consumer side, each message is consumed in a separate thread
but we currently have no way of waiting for all these threads to
complete.

In BaseRpcTestCase.test_call_timeout(), we complete the thread while
there is still a message handler running on the other side. This can
mean the consumer attempting to send its reply while other tests
are running, which clearly is a bad thing.

Make the connection close() and reset() methods in the kombu and qpid
drivers wait on proxy object threads before returning to avoid this
situation.

Change-Id: I8fb84b1dc7f31196ef27a5d1f8c4a3346b03e0ae
Merge "fix intermittent failures with deferred version test"
Merge "Don't use exclusive queues for fanout and direct"
Merge "Move rootwrap code to openstack.common"
Merge "Add a rpc envelope format version number."
Optionally add message envelope to notifications.

When message envelopes were added for rpc messages, notifications were
explicitly left out.  This was due to the fact that notifications are
consumed outside of OpenStack.  However, notification consumers will
likely want to eventually start taking advantage of some of the features
that the envelope is a prerequisite for, such as message signing.

This patch adds a new notification driver that produces message based
notifications including the message envelope.

Related to blueprint version-rpc-messages.

Change-Id: Iaa8b437e6b4f64053f4b02bf486516f60bb020e9
Fix timezone handling in timeutils tests

The tests for timeutils use some canned datetime values that include
timezones, and override the result of utcnow() with the same. This
is not correct because the real utcnow() returns naive datetime
objects, so tests that assume the contrary are incorrect.

This came about because I was unable to satisfy both the oslo unit
tests and the nova ones with the same code. That happened because
the oslo ones override utcnow() with an invalid result, and the
nova tests use the real one.

Change-Id: I683818f77ce0d3e3b2f07c0326d7a7196f3346c3
Sanitize cells calls.

Cells puts the data to be sanitized deeper into the message. We
should get this into oslo ready for when cells lands.

Change-Id: Ic5d111f11c979a179e0edcb314feb3f9d93f66ce
Enable millisecond logging by default

Related to Ic28867ae9a436e81c7f2fcf79f40a1ecc251072c

Change-Id: I135213cad3ec21edf8934c1094ee84ac93ed4561
Add common filter/filter handler for filter scheduler

Filter scheduler is being used for more than one core projects (Nova
and Cinder as of writing), the implementation shared a lot of common
code.  This patch moves base filter/filter handler class as well as
common filter implementation for filter scheduler into oslo to reduce
possible porting.

implement bp: common-filters

Change-Id: If0b1dee79c410c98e152230b55c1ec5dbcdef27c
Add common base weigher/weigher handler for filter scheduler

Filter scheduler is being used for more than one core projects (Nova
and Cinder as of writing), the implementation shared a lot of common
code. This patch is to move base weigher (weighing function), weigher
handler for filter scheduler into oslo to reduce possible porting.

implement bp: common-weights

Change-Id: I2d1b37438663b53e035cc262875a283e5e2ee970