Repository: AnalogJ/lexicon

Issue:
Rackspace provider not getting token from the identity API
_Update: solved! See my comment below._

I'm just starting out with lexicon and trying to get it working with Rackspace Cloud DNS. Requests are failing with a `requests.exceptions.HTTPError: 401 Client Error: Resource not found for validate token request` error. With debugging on, it appears lexicon is not requesting a token from `https://identity.api.rackspacecloud.com/v2.0/tokens` before it continues to submit a reqeuset to `https://dns.api.rackspacecloud.com/v1.0/…`.

Here's the command I'm running:

    lexicon rackspace --log_level DEBUG --auth-username myusername --auth-api-key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX list example.com A

And here's the output:

    Arguments: Namespace(action='list', auth_account=None, auth_api_key='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', auth_token=None, auth_username='myusername', content=None, delegated=None, domain='example.com', identifier=None, log_level='DEBUG', name=None, output='TABLE', priority=None, provider_name='rackspace', sleep_time=1, ttl=None, type='A')
    Starting new HTTPS connection (1): dns.api.rackspacecloud.com:443
    https://dns.api.rackspacecloud.com:443 "GET /v1.0/None/domains?name=example.com HTTP/1.1" 401 0
    Traceback (most recent call last):
      File "/Users/q/src/dns-lexicon/env/bin/lexicon", line 10, in <module>
        sys.exit(main())
      File "/Users/q/src/dns-lexicon/env/lib/python3.7/site-packages/lexicon/cli.py", line 117, in main
        results = client.execute()
      File "/Users/q/src/dns-lexicon/env/lib/python3.7/site-packages/lexicon/client.py", line 64, in execute
        self.provider.authenticate()
      File "/Users/q/src/dns-lexicon/env/lib/python3.7/site-packages/lexicon/providers/base.py", line 69, in authenticate
        return self._authenticate()
      File "/Users/q/src/dns-lexicon/env/lib/python3.7/site-packages/lexicon/providers/rackspace.py", line 66, in _authenticate
        'name': self.domain
      File "/Users/q/src/dns-lexicon/env/lib/python3.7/site-packages/lexicon/providers/base.py", line 142, in _get
        return self._request('GET', url, query_params=query_params)
      File "/Users/q/src/dns-lexicon/env/lib/python3.7/site-packages/lexicon/providers/rackspace.py", line 190, in _request
        response.raise_for_status()
      File "/Users/q/src/dns-lexicon/env/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 401 Client Error: Resource not found for validate token request for url: https://dns.api.rackspacecloud.com/v1.0/None/domains?name=example.com

Of course, this request will fail if is no token has been received from the `identity.api.rackspacecloud.com` API endpoint.

I'm happy to help troubleshoot, but I thought I'd ask first if I'm using the correct parameters for the rackspace provider. 


Discussion:
I found the cause of my original issue reported above. While setting up lexicon, I had exported this environment variable in my terminal, based on the example provided in the README.md:

    export LEXICON_RACKSPACE_TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

If this variable exists, the rackspace provider will not request a new token from the auth API. That's why that step was skipped, as shown in the debug output I posted above.

I unset this (and all the `LEXICON_*` env vars), so they were only provided via arguments on the command line. Now, the request to the auth API was completing successfully.

----

However, it still wasn't working even though I was now getting a response from the auth API, because:

1. Requests to the dns API did not include the tenant ID. Requests were going to `GET /v1.0/None/domains` instead of `GET /v1.0/XXXXXX/domains` (where `XXXXXX` is my tenant ID).
2. Requests to the dns API were not including a value with the `X-Auth-Token` header.

These two issues are solved by these code changes (I'll submit a pull request later):

```diff
--- rackspace-broken.py
+++ rackspace.py
@@ -60,6 +60,7 @@
                     }
                 }
             })
+            self._auth_account = auth_response['access']['token']['tenant']['id']
             self._auth_token = auth_response['access']['token']['id']

         payload = self._get('/domains', {
@@ -179,11 +180,11 @@
         if query_params is None:
             query_params = {}
         full_url = (self.api_endpoint +
-                    '/{0}' + url).format(self._get_provider_option('auth_account'))
+                    '/{0}' + url).format(self._auth_account)
         response = requests.request(action, full_url, params=query_params,
                                     data=json.dumps(data),
                                     headers={
-                                        'X-Auth-Token': self._get_provider_option('auth_token'),
+                                        'X-Auth-Token': self._auth_token,
                                         'Content-Type': 'application/json'
                                     })
```

This fix works for me because I will never want to provide a token as a config option. However, for those who still want to provide the tenant id and token via `_get_provider_option('…')`, this will require more work.
I'm not very familiar with the rackspace provider. The CODEOWNERS file mentions that it was written by @rmarscher Hopefully he has some ideas.

Is this issue tied to an alternative authentication mechanism that Rackspace supports? 
@quinncomendant Try with these environment variables:
export LEXICON_RACKSPACE_ACCOUNT=[tenant id]
export LEXICON_RACKSPACE_USERNAME=[username]
export LEXICON_RACKSPACE_API_KEY=[api key]

The cli options are defined here:
https://github.com/AnalogJ/lexicon/blob/v3.1.5/lexicon/providers/rackspace.py#L25-L40

The options from the provider are automatically set from corresponding environment vars in the format LEXICON_[provider]_[config argument] where the [config argument] is determined by removing `--auth-` from the front of the cli argument, replace `-` with `_` and make it uppercase.

Let me know if that doesn't work. I agree that this needs better documentation. @AnalogJ - a recommendation on where to place that would be great. I can help write it up when I get a moment.


Hello @rmarscher, just a little precision to say that now, removing `--auth` to get the environment variable name is deprecated. Now, it is simply `LEXICON_RACKSPACE_COMPLETE_CONFIG_NAME`.

So following variables can be used:
* `LEXICON_RACKSPACE_AUTH_ACCOUNT`
* `LEXICON_RACKSPACE_AUTH_USERNAME`
* `LEXICON_RACKSPACE_AUTH_API_KEY`

For documentation, my `auto` provider is gathering the options of all other providers. I will soon leverage this to build automatically a nice wiki page that gather all options.
OK. Thanks for letting me know, @adferrand.

@quinncomendant I'm testing out with my rackspace account and the latest master branch. There may be other fixes needed here along the lines of what you reported based on changes in lexicon since I last tested the provider (~April or May of 2018).



Resolving patch:
diff --git a/lexicon/providers/rackspace.py b/lexicon/providers/rackspace.py
--- a/lexicon/providers/rackspace.py
+++ b/lexicon/providers/rackspace.py
@@ -48,6 +48,16 @@ def __init__(self, config):
         self.api_endpoint = 'https://dns.api.rackspacecloud.com/v1.0'
         self.auth_api_endpoint = 'https://identity.api.rackspacecloud.com/v2.0'
         self._auth_token = None
+        self._auth_account = None
+
+    def _get_rackspace_option(self, key):
+        private_key = '_' + key
+        result = None
+        if hasattr(self, private_key):
+            result = getattr(self, private_key)
+        if result is None:
+            result = self._get_provider_option(key)
+        return result
 
     def _authenticate(self):
         self._auth_token = self._get_provider_option('auth_token')
@@ -61,6 +71,7 @@ def _authenticate(self):
                 }
             })
             self._auth_token = auth_response['access']['token']['id']
+            self._auth_account = auth_response['access']['token']['tenant']['id']
 
         payload = self._get('/domains', {
             'name': self.domain
@@ -178,12 +189,13 @@ def _request(self, action='GET', url='/', data=None, query_params=None):
             data = {}
         if query_params is None:
             query_params = {}
+        LOGGER.debug('request tenant ID: %s', self._get_rackspace_option('auth_account'))
         full_url = (self.api_endpoint +
-                    '/{0}' + url).format(self._get_provider_option('auth_account'))
+                    '/{0}' + url).format(self._get_rackspace_option('auth_account'))
         response = requests.request(action, full_url, params=query_params,
                                     data=json.dumps(data),
                                     headers={
-                                        'X-Auth-Token': self._get_provider_option('auth_token'),
+                                        'X-Auth-Token': self._get_rackspace_option('auth_token'),
                                         'Content-Type': 'application/json'
                                     })
         # if the request fails for any reason, throw an error.
@@ -194,7 +206,7 @@ def _request(self, action='GET', url='/', data=None, query_params=None):
     def _request_and_wait(self, action='POST', url='/', data=None, query_params=None):
         result = self._request(action, url, data, query_params)
 
-        sleep_time = self._get_provider_option('sleep_time') or '1'
+        sleep_time = self._get_rackspace_option('sleep_time') or '1'
         sleep_time = float(sleep_time)
 
         while not _async_request_completed(result):
@@ -222,7 +234,7 @@ def _update_response(self, payload):
         response = requests.request('GET', payload['callbackUrl'], params={'showDetails': 'true'},
                                     data={},
                                     headers={
-                                        'X-Auth-Token': self._get_provider_option('auth_token'),
+                                        'X-Auth-Token': self._get_rackspace_option('auth_token'),
                                         'Content-Type': 'application/json'})
 
         # if the request fails for any reason, throw an error.


Test patch: