make record_metering_data concurrency safe

Recording metering might happen in parallel, and if two sqlalchemy
session try to create a source, a user, a project or a resource at the
same time, one of them fail with a IntegrityError.

This patch ensure the record_metering_data sqlalchemy session always
have the needed source, user, project and a resource by creating missing
obj in an other sqlalchemy session.

Fixes bug: #1237671

Change-Id: Ie8761ce1615c3142ad6ad2a400537f465f3bbf0a
(cherry picked from commit dd62f9ce0d8d5abac0fb34b0520515b6c4566c1f)
Avoid intermittent integrity error on alarm creation

Fixes bug 1255107

Previously, alarm created failed intermittently with the
sqlalchemy driver, due to the reliance on session merge to create
the project & user IDs if not already encountered.  This approach
conflicted with the strict foreign key constraints that were
added to the alarms table as an after-thought.

Now, we avoid the potential problem by creating the project and
user IDs explicitly.

Unfortunately, this cannot be tested via the scenario tests,
as there we use sqlite (the problematic migration does not
apply the foreign key constraints for that engine).

Change-Id: Ieaed676a0a68725242fadbf658646b874d6851ce
(cherry picked from commit b2b500c83cf5a81269c84a16f9a30ab7fa4c8ee6)
Merge "make record_metering_data concurrency safe" into stable/havana
Merge "Avoid intermittent integrity error on alarm creation" into stable/havana
Properly reconnect subscribing clients when QPID broker restarts

Fixes bug 1251757

This is a cherrypick of oslo-incubator:
55678c7ab7fbc1b2beb428d046bb4162f23b0a53

From the oslo-incubator bug:
When the QPID broker is restarted (or fails over), subscribed clients
will attempt to re-establish their connections.  In the case of fanout
subscriptions, this reconnection functionality is broken. For version
1 topologies, the clients attempt to reconnect twice to the same
exclusive address - which is illegal.  In the case of version 2
topologies, the address parsing is broken and an illegal address is
created on reconnect.  This fix avoids the problem by removing the
special-case reconnect code that manages UUID addresses; it is
unnecessary as the QPID broker will generate unique queue names
automatically when the clients reconnect.

Change-Id: If966bb25e49b5837293a1bb181bbb02086599d81
Create a shared queue for QPID topic consumers

Fixes bug 1257293

This is a cherrypick of oslo-incubator:
e355fa3a9f49ff524ce98d1263a784b5a7929dfb

From the oslo-incubator commit:
When multiple RPC servers (consumers) are subscribed to the same RPC
topic, a single RPC request to that topic should be received by only
one of the consumers.  A bug in the QPID driver caused every consumer
to receive a copy of the RPC request.  This bug affects only Topology
version 2.  This patch will cause a single queue to be created for
each topic, and shared among all consumers of that topic.  This
results in each RPC request being received by only one consumer,
in turn across all the competing consumers.

Change-Id: I97c6eb5fe63322ab70716c63074818ff1b6ea82b
Merge "Properly reconnect subscribing clients when QPID broker restarts" into stable/havana
Merge "Create a shared queue for QPID topic consumers" into stable/havana
Updated from global requirements

Change-Id: I3c2bf6087cf07cddb4b1290470d5180edbbec42f
Bump stable/havana next version to 2013.2.2

Change-Id: I3a7170180717bd62fce4cbc2c3a9107187f89fb1
Avoid false negatives on message signature comparison

Fixes bug 1262255

Previously, samples for the 'instance.scheduled' meter were always
rejected with a false negative on the message signature verification.

The problem occured because certain resource metadata (such as the
block_device_mapping) in the 'scheduler.run_instance.scheduled'
notification are realized as a list of dict, and are thus similarly
encoded in the corresponding sample payload.

However the message signature computation then became indeterminate,
as the unicode representation of equivalent dicts is not guaranteed
equal (e.g. depends on insertion order in the case of hash collisions).

Now, we avoid false negatives by explicitly re-ordering insertion
into such dicts.

Change-Id: I77f7d89229518cf040608d7eb3307e2257bce07a
(cherry picked from commit e9cde137b0c24ac755bbb3629b9a831391574f6c)
1st & last sample timestamps in Resource representation

Fixes bug 1262335

Previously, the first and last sample timestamps associated
with a resource were calculated by the storage driver, but
then discarded.

Now, we include these values in the API resource representation.

Change-Id: I3cf6081e1ac1294083f6326f85dabebab0d02203
(cherry picked from commit 6a9dc97f962c288f5037d59d05a771823eb4bcdd)
Merge "Avoid false negatives on message signature comparison" into stable/havana
Remove redundant meter (name,type,unit) tuples from Resource model

Fixes bug 1267162

Previously, a list of meter (name,type,unit) tuples with per-sample
cardinality was added to the Resource model by the storage driver
get_resource method.

However this potentially huge and highly repetitive list was then
promptly discarded by the API layer, as the meter links in the API
Resource representation are populated via subsequent calls into
the storage driver get_meters method.

Now, we avoid the scaling issue by suppressing the creation of
this unused meter list.

Change-Id: I18dee7552836c724798ea1340e2e933c09d8a294
(cherry picked from commit 7c4c0e8fc098fbadec2e30ea2bdf1068b5b4288a)
Update ceilometer.conf.sample

Due to python-keystoneclient update.

Change-Id: Ie683d3fbf8086d2be6a2fbe19374b8d31abaf976
Closes-bug: #1268614
(cherry picked from commit 76c06e03eaef0c66689c62aece24baee0b372644)
Replace mongo aggregation with plain ol' map-reduce

Fixes bug 1262571

Previously, the mongodb storage driver an aggregation pipeline
over the meter collection in order to construct a list of resources
adorned with first & last sample timestamps etc.

However mongodb aggregation framework performs sorting in-memory,
in this case operating over a potentially very large collection.
It is also hardcoded to abort any sorts in an aggregation pipeline
that will consume more than 10% of physical memory, which is
observed in this case.

Now, we avoid the aggregation framework altogether and instead
use an equivalent map-reduce.

Change-Id: Ibef4a95acada411af385ff75ccb36c5724068b59
(cherry picked from commit ba6641afacfc52e7391d2095751ee96d62a64c25)
Fix the Alarm documentation of Web API V2

Correct the Alarm example on the API documentation to contain a
valid Alarm sample and complete the Note section with information
about the connection between the type and rules fields of the
Alarm.

Fixes bug #1245362

Change-Id: I5fccf51b820330595a627fd0001beec2d5f7c6e3
Fix to tackle instances without an image assigned

fixes bug #1233103

Change-Id: I4d4d7bdff09b6748e14ebac8293050513fb18ed2
(cherry picked from commit e36f9f32fea2e128e1f03bf2b94ad0584337b2c4)
Merge "Replace mongo aggregation with plain ol' map-reduce" into stable/havana
Merge "Fix the Alarm documentation of Web API V2" into stable/havana