Allow any attributes in mapping and include in results

When using attribute mapping (e.g., user_additional_attribute_mapping), any
attributes that Keystone didn't know about were ignored and a warning was
issued, like

 WARNING keystone.common.ldap.core [-] Invalid additional attribute mapping:
 "description:description". Value "description" must use one of password,
 enabled, default_project_id, name, email.

This check is removed. Additional attribute mappings for "unexpected"
attributes (such as description) should be allowed. For one thing, all the
known attributes already have a way of setting the mapping, via the
user_*_attribute config option. This will allow setting a "description" or
any other attribute mapping for users.

The extra attributes defined in the *_additional_attribute_mapping config
options weren't included in the list results. These attributes should be
included in the list results.

This is a combination of 3 commits from master:
(cherry picked from commit 17db03164200f0d7fff564058fc7df8e97705523)
(cherry picked from commit 10051adace7cd728d46cd4c999e3019fdbf2d961)
(cherry picked from commit ebfdab034f2b35074f1645b70c1f07d0c4dfd337)

Change-Id: Ia80174063a5eacdef690ca378b7cb71300a5f5f8
Closes-Bug: #1293698
Removed duplication with list_user_ids_for_project

Keystone list_user_ids_for_project has the possibility of returning
duplicate users if the user has more than one role in the project.
list_user_ids_for_project queries the assignment table which has
one entry for every role the user has but will now remove
duplicate combinations of target_id and actor_id. This impacts
V2 calls for tenants.list_users(<tenant_id>) and
users.list(tenant_id=<tenant_id>).

Change-Id: I9a1e5b944c2c627138851976e81d405f54fae1c9
Closes-Bug: #1308218
(cherry picked from commit d6cb7043cb47f5e73b61635af596736314441275)
Updated from global requirements

Change-Id: Ieea9cc5b41ac787fe9b1378f0df2f5c43ad4e6a9
SQL and LDAP fixes for get_roles_for_user_and_project user=group ID

When there was a role assigned to a group with the same ID as a user,
the SQL and LDAP assignment backends would incorrectly return the
assignment to the group when requesting roles for the user via the
get_roles_for_user_and_project method.

With this change, assignments to a group with the same ID are not
returned for the user when calling get_roles_for_user_and_project.

Functions were added to compare DNs more accurately based on the
LDAP RFCs.

The fakeldap code was changed to normalize the values when
comparing values for checking if the values match the filter.

Co-Authored By: Nathan Kinder <nkinder@redhat.com>
Co-Authored By: Adam Young <ayoung@redhat.com>

Change-Id: Id3d6f66c995e65e37d909359420d71ecdde86b69
Closes-Bug: #1309228
sql migration: ensure using innodb utf8 for assignment table

During migration from 37 to 38, migration fails to create assignment
table if we don't explictly specify InnoDB and UTF8 charset.

Closes-bug #1320855
Signed-off-by: Emilien Macchi <emilien.macchi@enovance.com>

Change-Id: Icd88202d2f020aea02514e49c2d07e349c74949f
(cherry picked from commit 18289b9e26ba900302874e72fcbf4529580a973a)
Merge "Allow any attributes in mapping and include in results" into stable/icehouse
Merge "SQL and LDAP fixes for get_roles_for_user_and_project user=group ID" into stable/icehouse
Merge "Removed duplication with list_user_ids_for_project" into stable/icehouse
Merge "sql migration: ensure using innodb utf8 for assignment table" into stable/icehouse
Encode/Decode LDAP parameters to/from UTF-8

The python-ldap API only accepts UTF-8 encoded strings therefore any
unicode values must be encoded to UTF-8 prior to passing to
python-ldap and conversely UTF-8 encoded strings returned from
python-ldap need to be decoded back from UTF-8 into unicode.

Need to use unicode() rather than str() to properly handle non-ascii
characters, but to be PY2/PY3 compatible use six.text_type.

Very few of the ldap tests were using non-ascii values, in fact
non-ascii values were restricted to only specific tests that
had 'unicode' in their test name. This is very weak test coverage.
Replace all occurances of 'fake', the standard string
used in the tests for test value with 'fäké' where the a has an
umlaut and the e has a diacritical. Visually they look almost the
same but will trigger the type of encoding exceptions we've seen
in the past.

This is the minimal backport for icehouse from the following
master commits:
https://review.openstack.org/#/c/82396/
https://review.openstack.org/#/c/82398/
https://review.openstack.org/#/c/82399/

Closes-Bug: #1172106

Change-Id: I6b328dfc8435457a8d7ff16320f3d869cfa1043c
Block delegation escalation of privilege

Forbids doing the following with either a trust
  or oauth based token:
  creating a trust
  approving a request_token
  listing request tokens

Change-Id: I1528f9dd003f5e03cbc50b78e1b32dbbf85ffcc2
Closes-Bug:  1324592
Merge "Encode/Decode LDAP parameters to/from UTF-8" into stable/icehouse
Make sure domains are enabled by default

Domains without an explicit 'enabled' key in the dict will now be
handled in the same way as projects are. This will ensure that
the KVS backend produces the same result as the SQL backend for
assignment.

Change-Id: I3b6d26278329841cfe1be5ce7f67ba6d3569785f
Related-Bug: #1311142
Properly invalidate cache for get_*_by_name methods

Use the original name for invalidating the cache on the
get_project_by_name and get_domain_by_name when performing update
operations on the name attributes.

Change-Id: If58a3b0a47501096f9fa9eb9cd7a6057f7b29298
Closes-Bug: #1311142
Bump stable/icehouse next version to 2014.1.2

Change-Id: Id7181755d3609c0ae91c2402b34f2de02c06a1f0
Merge "Make sure domains are enabled by default" into stable/icehouse
Merge "Properly invalidate cache for get_*_by_name methods" into stable/icehouse
Ensure that in v2 auth tenant_id matches trust

Previously if a trustee requests a trust scoped token for a project that
is different to the one in the trust, however the trustor has the
appropriate roles then a token would be issued.

Ensure that the trust that was given matches the project that was
specified in the scope.

(cherry picked from commit 1556faec2f65dba60584f0a9657d5b717a6ede3a)

Change-Id: I00ad783bcb93cea9e5622965f81b91c80f4570cc
Closes-Bug: #1331912
HEAD responses should return same status as GET

According to the HTTP spec, a HEAD request should return the same
status and headers as the GET request (including content-type and
content-length). The HEAD request simply strips out the body and
returns no body. Any case where HEAD routing returned a different
status code than GET, now returns the same status and headers.

Any case where HEAD was supported where GET was not supported now
supports both GET and HEAD.

The wsgi.render_response code now handles HEAD appropriately and
will maintain headers while enforcing no body data is returned.

The bulk of this change is to support the same behavior between
deploying Keystone under eventlet and under HTTPD + mod_wsgi. In
the case of deploying under HTTPD + mod_wsgi, there are cases
where mod_wsgi will turn a HEAD request into a GET request to
ensure that the proper response is rendered. With these changes
all HEAD responses will respond in the same manner under either
eventlet or mod_wsgi.

Change-Id: I13ce159cbe9739d4bf5d321fc4bd069245f32734
Closes-Bug: #1334368
(cherry picked from commit 82101a36d348ad0c3de10f455878828f4263ffe2)