feat(validation): validate user input

The validation is implemented as a set of free-standing functions
in a separated module; each driver which uses storage functions
should call call the corresponding validation functions first.

Fixes: bug #1177971
Fixes: bug #1177976
Fixes: bug #1177979
Fixes: bug #1206175
Fixes: bug #1208873

Change-Id: I0b9c9766a9c3de42f2a2eb470cba0ab7265e669a
Use oslo.config 1.2

Some important fixes were released in version 1.2, it should be safe to
upgrade the version Marconi's using.

Change-Id: Idd63a23347f5e9bb91597d76068439846ee2e2cb
Merge "Use oslo.config 1.2"
doc(validation): update the sample config file

This patch add the configuration variables of transport driver-
specific limits and input validation to the sample config file,
so that the users don't need to open the source code to figure
out how to change the limits :)

Change-Id: I811b7dc4ca44d25a3cdb5402e11d599aa532ab39
Fixes unique constraint regex

This patch fixes the regex being used to extract the marker from the
unique index error. The issue was caused because we introduced project
to the queue_marker index and it wasn't being considered in that regex.

This regex is more general. It matches the index name and gets the
marker from the end of the error message.

Change-Id: I02bc153636f10bc49b8634fc4bfeb4bf2b6d6597
Closes-bug: #1211386
fix(storage/mongo): unclaimed messages not listed

Adds a unit test that reproduces this error and applies the fix.

The fix is subtle. When messages are unclaimed, their claim expiration
time and claim ID are set to 0 and None respectively. The cause for
the bug is that claim expiration time is set to 0, rather than a valid
datetime object. As a result, the comparison silently fails.

The fix: fast-forward the expiration time of a claim to now.

Change-Id: I7781795c394b2a471012e072a522ee228d56a894
Closes-Bug: #1210633
Merge "Fixes unique constraint regex"
Convert "outgoing" literals to unicode

The patch convers most literals throughout marconi to unicode. Lietrals
not leaving "marconi-land" were left untouched.

This is the first patch of a serie that will enforce unicode usage
within marconi. Decode early, encode late.

String format parameters were left untouched on purpose. Those
parameters should be decoded before getting there.

Implements blueprint enforce-decoding

Change-Id: I85e534ced188191c9c7a17e9908cb720e7d63ca9
feat(validation): check metadata JSON size

This patch also factor the de-serialized validation tests into a
separated module (which fixes the tests for mongo driver).

Change-Id: I46f0ac7df06ad2bf455073347392f7f4159449c9
Merge "feat(validation): check metadata JSON size"
fix: rename _payload_uplimit into _paging_uplimit

Although "message_paging_uplimit" also limit the maximum number of
IDs can be supplied in a URI, which is not quite a "page" (in bulk
deletion), but we don't need the configuration to be too precise.

Change-Id: I0737146f1212c82db18de35e35206d3932a46628
Don't use claim creation time to calc claim's age

Mongodb's driver uses Claim's creation time to estimate claims age
instead, it should use claim's update time. Since there's no update /
creation time in mongodb's claims, the patch estimates claims age by
doing:

    delta(expiration - ttl, now)

This change fixes the issue bellow and makes mongodb's claims compliant
with Marconi's specs.

Change-Id: I72c968236ecd239e7b885b293238e2920807a57c
Closes-Bug: #1210662
Merge "Don't use claim creation time to calc claim's age"
Ignore *.egg dirs

This patch adds *.egg to .gitignore

Change-Id: Ieb2ff6bc04a101fb700419ccfabbc947725de481
fix: fetching nonexistent message/claims raises 503s

The proposed fix moves away from throwing an exception everytime a
malformed ID is encountered and instead returns None. This way,
partial bulk get and bulk deletes are possible.

Storage drivers affected:
- mongo
- sqlite

Changes/expected behavior:
- GET /v1/queues/exists/messages/malformed => 404
- GET /v1/queues/exists/messages?ids=malformed,malformed => 204
- DELETE /v1/queues/exists/messages?ids=malformed,malformed => 204
- GET /v1/queues/exists/claims/malformed => 404
- PATCH /v1/queues/exists/claims/malformed => 404
- DELETE /v1/queues/exists/claims/malformed => 204

In partcular, regarding bulk_get on messages, malformed IDs or
messages that are not found by ID are now ignored. bulk_delete also
works in this fashion now.

Regression tests are added to the unit test suite to reflect these
changes.

Change-Id: I4f14cd8b4cfd1dd190dccd8724c20f5fac99c629
Closes-Bug: #1203842
test: delete msg w/ invalid claim isn't deleted

Adds a test to verify the above scenario. It should be returning 204
in this case without the message being affected.

Change-Id: I74cf8ab5f7d94128ae2157a51a5e37c01ef9ca64
restructure: rm -rf tools, move test requires

To pave the path for the project structure of the future, tools/ is
being eliminated.

New commands/utils should be added as entry points and placed in cmd/

tools/system-test-requires -> system-test-requirements.txt

Change-Id: I14b3189f2b2a1cee6508952a46274e00514addac
Use req.get_param's store feature in lieu of utils.purge

This patch removes the use of utils.purge since it is no
longer needed in light of Falcon's relatively new "store"
feature, which causes the target dict to only receive the
value of the param if it is found in the request, thus
avoiding clobbering default kwarg values, which was the
original intent of utils.purge.

As a bonus, this approach is also ~40% faster.

Change-Id: I6b3d82b1f700045e50b3131902b2c7b6dc6ea13c
Implements: blueprint v1-obvious-optimizations
Don't wait for ack when deleting expired messages

This patch simply sets write concern to 0 in the MongoDB driver,
when removing expired messages. This shortens the amount of time
that garbage collection takes, and slightly reduces the amount
of traffic going between app and db servers, as well as between
master and secondary nodes.

This is a safe thing to do, since the operator will be running
multiple GC daemons, so that in the case of a failure, the
operation will be retried in the near future.

The downside is that persistent failures to collect expired
messages will cause the DB to grow out of control; however,
it is expected that if this operation is failing, more than
likely message CRUD in general is failing, which mitigates
the risk just mentioned.

Change-Id: I90950721361dacffa508624281d2e2ea8cf53724
Merge "Ignore *.egg dirs"