Repository: cevoaustralia/aws-google-auth

Issue:
Cannot login: MFA stopped working, then took wrong choice
I signed in no problem on 2020-10-12, but today, 2020-10-13 I cannot sign in because MFA does not work. Looks like Google changed something. Note that `saml2aws` also [stopped working](https://github.com/Versent/saml2aws/issues/568).

Here are the requests (parameters omitted) the Debug log shows are being made:

```
GET /o/saml2/initsso
GET /ServiceLogin
POST /signin/v1/lookup 
GET /signin/challenge/pwd/1
POST /signin/challenge/pwd/1 
GET /signin/selectchallenge/2
```
At this point `aws-google-auth` gives me this prompt:
```
Choose MFA method from available:

Enter MFA choice number (9): 
```
Entering 9 (or anything else) gives me a `KeyError`.

Running the same command, sometimes I get other prompts. For example:
```
Choose MFA method from available:
4: OOTP (Google Prompt)
Enter MFA choice number (9): 4
MFA Type Chosen: OOTP (Google Prompt)
MFA token: 
```
Here I selected 4, but never received the Google Prompt on my phone.

After several attempts, getting only the above prompts, I finally got this prompt:
```
Choose MFA method from available:
3: SMS
4: OOTP (Google Prompt)
Enter MFA choice number (9): 4
MFA Type Chosen: OOTP (Google Prompt)
Enter SMS token: G-
```
Note that I chose OOTP, but then was prompted for (and received) an SMS token, and entering the SMS token authenticated me. Further API calls were:
```
POST /signin/challenge/ipp/4
GET /signin/challenge/ipp/4
# Here is where I entered my SMS token
POST /signin/challenge/ipp/4
GET /o/saml2/initsso
```
#### Update 1
Trying several more times, I did get to a prompt that was only SMS codes, and entering 3 (the displayed choice) yielded
```
MFA Type Chosen: SMS
Something went wrong - Could not find SAML response, check your credentials or use --save-failure-html to debug.
```
Selecting 4 when only SMS was shown as an option yielded a KeyError.

I added backup codes to my account hoping that would help. Interesting that it made the default MFA choice number when no options were provided increase from 9 to 10. After that, and a few more attempts, I got:
```
Please visit the following URL to view your CAPTCHA: https://accounts.google.com/Captcha?...
Captcha (case insensitive): unscuryar
Choose MFA method from available:
4: OOTP (Google Prompt)
Enter MFA choice number (10): 4
MFA Type Chosen: OOTP (Google Prompt)
ERROR:root:SAML lookup failed, storing failure page to 'saml.html' to assist with debugging.
Something went wrong - Could not find SAML response, check your credentials or use --save-failure-html to debug.
```
The saved saml.html showed that the page was asking for a backup code.

Discussion:

Patch:
diff --git a/README.rst b/README.rst
index 59c258e..1fe975f 100644
--- a/README.rst
+++ b/README.rst
@@ -119,7 +119,7 @@ Usage
                            [--bg-response BG_RESPONSE]
                            [--saml-assertion SAML_ASSERTION] [--no-cache]
                            [--print-creds] [--resolve-aliases]
-                           [--save-failure-html] [-a | -r ROLE_ARN] [-k]
+                           [--save-failure-html] [--save-saml-flow] [-a | -r ROLE_ARN] [-k]
                            [-l {debug,info,warn}] [-V]
 
     Acquire temporary AWS credentials via Google SSO
@@ -151,6 +151,7 @@ Usage
       --resolve-aliases     Resolve AWS account aliases.
       --save-failure-html   Write HTML failure responses to file for
                             troubleshooting.
+      --save-saml-flow      Write all GET and PUT requests and HTML responses to/from Google to files for troubleshooting.
       -a, --ask-role        Set true to always pick the role
       -r ROLE_ARN, --role-arn ROLE_ARN
                             The ARN of the role to assume ($AWS_ROLE_ARN)
@@ -260,11 +261,11 @@ by this tool are:
 +==================+=====================================+
 | No second factor | (none)                              |
 +------------------+-------------------------------------+
-| TOTP (eg Google  | ``.../signin/challenge/totp/2?...`` |
+| TOTP (eg Google  | ``.../signin/challenge/totp/...`` |
 |  Authenticator   |                                     |
 |  or Authy)       |                                     |
 +------------------+-------------------------------------+
-| SMS (or voice    | ``.../signin/challenge/ipp/2?...``  |
+| SMS (or voice    | ``.../signin/challenge/ipp/...``  |
 |  call)           |                                     |
 +------------------+-------------------------------------+
 | SMS (or voice    | ``.../signin/challenge/iap/...``    |
@@ -272,7 +273,7 @@ by this tool are:
 |  number          |                                     |
 |  submission      |                                     |
 +------------------+-------------------------------------+
-| Google Prompt    | ``.../signin/challenge/az/2?...``   |
+| Google Prompt    | ``.../signin/challenge/az/...``   |
 |  (phone app)     |                                     |
 +------------------+-------------------------------------+
 | Security key     | ``.../signin/challenge/sk/...``     |
diff --git a/aws_google_auth/__init__.py b/aws_google_auth/__init__.py
index 14cc291..107f974 100644
--- a/aws_google_auth/__init__.py
+++ b/aws_google_auth/__init__.py
@@ -41,6 +41,7 @@ def parse_args(args):
     parser.add_argument('--print-creds', action='store_true', help='Print Credentials.')
     parser.add_argument('--resolve-aliases', action='store_true', help='Resolve AWS account aliases.')
     parser.add_argument('--save-failure-html', action='store_true', help='Write HTML failure responses to file for troubleshooting.')
+    parser.add_argument('--save-saml-flow', action='store_true', help='Write all GET and PUT requests and HTML responses to/from Google to files for troubleshooting.')
 
     role_group = parser.add_mutually_exclusive_group()
     role_group.add_argument('-a', '--ask-role', action='store_true', help='Set true to always pick the role')
@@ -238,7 +239,7 @@ def process_auth(args, config):
         # Validate Options
         config.raise_if_invalid()
 
-        google_client = google.Google(config, args.save_failure_html)
+        google_client = google.Google(config, save_failure=args.save_failure_html, save_flow=args.save_saml_flow)
         google_client.do_login()
         saml_xml = google_client.parse_saml()
         logging.debug('%s: saml assertion is: %s', __name__, saml_xml)
diff --git a/aws_google_auth/_version.py b/aws_google_auth/_version.py
index acac8db..2b6a10a 100644
--- a/aws_google_auth/_version.py
+++ b/aws_google_auth/_version.py
@@ -1,1 +1,1 @@
-__version__ = "0.0.36"
+__version__ = "0.0.37"
diff --git a/aws_google_auth/google.py b/aws_google_auth/google.py
index c7f641c..7ddcd1c 100644
--- a/aws_google_auth/google.py
+++ b/aws_google_auth/google.py
@@ -12,6 +12,7 @@ import sys
 
 import requests
 from PIL import Image
+from datetime import datetime
 from distutils.spawn import find_executable
 from bs4 import BeautifulSoup
 from requests import HTTPError
@@ -34,7 +35,7 @@ class ExpectedGoogleException(Exception):
 
 
 class Google:
-    def __init__(self, config, save_failure):
+    def __init__(self, config, save_failure, save_flow=False):
         """The Google object holds authentication state
         for a given session. You need to supply:
 
@@ -52,6 +53,11 @@ class Google:
         self.base_url = 'https://accounts.google.com'
         self.save_failure = save_failure
         self.session_state = None
+        self.save_flow = save_flow
+        if save_flow:
+            self.save_flow_dict = {}
+            self.save_flow_dir = "aws-google-auth-" + datetime.now().strftime('%Y-%m-%dT%H%M%S')
+            os.makedirs(self.save_flow_dir, exist_ok=True)
 
     @property
     def login_url(self):
@@ -89,9 +95,35 @@ class Google:
 
         return sess
 
-    def post(self, url, data=None, json=None):
+    def _save_file_name(self, url):
+        filename = url.split('://')[1].split('?')[0].replace("accounts.google", "ac.go").replace("/", "~")
+        file_idx = self.save_flow_dict.get(filename, 1)
+        self.save_flow_dict[filename] = file_idx + 1
+        return filename + "_" + str(file_idx)
+
+    def _save_request(self, url, method='GET', data=None, json_data=None):
+        if self.save_flow:
+            filename = self._save_file_name(url) + "_" + method + ".req"
+            with open(os.path.join(self.save_flow_dir, filename), 'w', encoding='utf-8') as out:
+                try:
+                    out.write("params=" + url.split('?')[1])
+                except IndexError:
+                    out.write("params=None")
+                out.write(("\ndata: " + json.dumps(data, indent=2)).replace(self.config.password, '<PASSWORD>'))
+                out.write(("\njson: " + json.dumps(json_data, indent=2)).replace(self.config.password, '<PASSWORD>'))
+
+    def _save_response(self, url, response):
+        if self.save_flow:
+            filename = self._save_file_name(url) + ".html"
+            with open(os.path.join(self.save_flow_dir, filename), 'w', encoding='utf-8') as out:
+                out.write(response.text)
+
+    def post(self, url, data=None, json_data=None):
         try:
-            response = self.check_for_failure(self.session.post(url, data=data, json=json))
+            self._save_request(url, method='POST', data=data, json_data=json_data)
+            response = self.check_for_failure(self.session.post(url, data=data, json=json_data))
+            self._save_response(url, response)
+
         except requests.exceptions.ConnectionError as e:
             logging.exception(
                 'There was a connection error, check your network settings.', e)
@@ -108,7 +140,10 @@ class Google:
 
     def get(self, url):
         try:
+            self._save_request(url)
             response = self.check_for_failure(self.session.get(url))
+            self._save_response(url, response)
+
         except requests.exceptions.ConnectionError as e:
             logging.exception(
                 'There was a connection error, check your network settings.', e)
@@ -162,7 +197,7 @@ class Google:
     @staticmethod
     def find_app_id(inputString):
         try:
-            searchResult = re.search('"appid":"[a-z://.-_]+"', inputString).group()
+            searchResult = re.search('"appid":"[a-z://.-_] + "', inputString).group()
             searchObject = json.loads('{' + searchResult + '}')
             return str(searchObject['appid'])
         except:
@@ -176,45 +211,31 @@ class Google:
 
         # Collect information from the page source
         first_page = BeautifulSoup(sess.text, 'html.parser')
-        gxf = first_page.find('input', {'name': 'gxf'}).get('value')
+        # gxf = first_page.find('input', {'name': 'gxf'}).get('value')
         self.cont = first_page.find('input', {'name': 'continue'}).get('value')
-        page = first_page.find('input', {'name': 'Page'}).get('value')
-        sign_in = first_page.find('input', {'name': 'signIn'}).get('value')
-        account_login_url = first_page.find('form', {'id': 'gaia_loginform'}).get('action')
+        # page = first_page.find('input', {'name': 'Page'}).get('value')
+        # sign_in = first_page.find('input', {'name': 'signIn'}).get('value')
+        form = first_page.find('form', {'id': 'gaia_loginform'})
+        account_login_url = form.get('action')
 
-        payload = {
-            'bgresponse': 'js_disabled',
-            'checkConnection': '',
-            'checkedDomains': 'youtube',
-            'continue': self.cont,
-            'Email': self.config.username,
-            'gxf': gxf,
-            'identifier-captcha-input': '',
-            'identifiertoken': '',
-            'identifiertoken_audio': '',
-            'ltmpl': 'popup',
-            'oauth': 1,
-            'Page': page,
-            'Passwd': '',
-            'PersistentCookie': 'yes',
-            'ProfileInformation': '',
-            'pstMsg': 0,
-            'sarp': 1,
-            'scc': 1,
-            'SessionState': '',
-            'signIn': sign_in,
-            '_utf8': '?',
-        }
+        payload = {}
+
+        for tag in form.find_all('input'):
+            if tag.get('name') is None:
+                continue
+
+            payload[tag.get('name')] = tag.get('value')
+
+        payload['Email'] = self.config.username
 
         if self.config.bg_response:
             payload['bgresponse'] = self.config.bg_response
 
-        # GALX is sometimes not there
-        try:
-            galx = first_page.find('input', {'name': 'GALX'}).get('value')
-            payload['GALX'] = galx
-        except:
-            pass
+        if payload.get('PersistentCookie', None) is not None:
+            payload['PersistentCookie'] = 'yes'
+
+        if payload.get('TrustDevice', None) is not None:
+            payload['TrustDevice'] = 'on'
 
         # POST to account login info page, to collect profile and session info
         sess = self.post(account_login_url, data=payload)
@@ -405,35 +426,18 @@ class Google:
         newPayload = {}
 
         auth_response_page = BeautifulSoup(response.text, 'html.parser')
+        form = auth_response_page.find('form')
+        for tag in form.find_all('input'):
+            if tag.get('name') is None:
+                continue
 
-        challengeId = auth_response_page.find('input', {
-            'name': 'challengeId'
-        }).get('value')
-        challengeType = auth_response_page.find('input', {
-            'name': 'challengeType'
-        }).get('value')
-        tl = auth_response_page.find('input', {
-            'name': 'TL'
-        }).get('value')
-        gxf = auth_response_page.find('input', {
-            'name': 'gxf'
-        }).get('value')
+            newPayload[tag.get('name')] = tag.get('value')
+
+        newPayload['Email'] = self.config.username
+        newPayload['Passwd'] = self.config.password
 
-        newPayload['challengeId'] = challengeId
-        newPayload['challengeType'] = challengeType
-        newPayload['TL'] = tl
-        newPayload['gxf'] = gxf
-        newPayload['TrustDevice'] = "on"
-        newPayload['continue'] = payload['continue']
-        newPayload['ltmpl'] = payload['ltmpl']
-        newPayload['scc'] = payload['scc']
-        newPayload['sarp'] = payload['sarp']
-        newPayload['checkedDomains'] = payload['checkedDomains']
-        newPayload['checkConnection'] = payload['checkConnection']
-        newPayload['pstMsg'] = payload['pstMsg']
-        newPayload['oauth'] = payload['oauth']
-        newPayload['Email'] = payload['Email']
-        newPayload['Passwd'] = payload['Passwd']
+        if newPayload.get('TrustDevice', None) is not None:
+            newPayload['TrustDevice'] = 'on'
 
         return self.post(response.url, data=newPayload)
 
@@ -533,48 +537,23 @@ class Google:
 
         sms_token = input("Enter SMS token: G-") or None
 
-        payload = {
-            'challengeId':
-            response_page.find('input', {
-                'name': 'challengeId'
-            }).get('value'),
-            'challengeType':
-            response_page.find('input', {
-                'name': 'challengeType'
-            }).get('value'),
-            'continue':
-            response_page.find('input', {
-                'name': 'continue'
-            }).get('value'),
-            'scc':
-            response_page.find('input', {
-                'name': 'scc'
-            }).get('value'),
-            'sarp':
-            response_page.find('input', {
-                'name': 'sarp'
-            }).get('value'),
-            'checkedDomains':
-            response_page.find('input', {
-                'name': 'checkedDomains'
-            }).get('value'),
-            'pstMsg':
-            response_page.find('input', {
-                'name': 'pstMsg'
-            }).get('value'),
-            'TL':
-            response_page.find('input', {
-                'name': 'TL'
-            }).get('value'),
-            'gxf':
-            response_page.find('input', {
-                'name': 'gxf'
-            }).get('value'),
-            'Pin':
-            sms_token,
-            'TrustDevice':
-            'on',
-        }
+        challenge_form = response_page.find('form')
+        payload = {}
+        for tag in challenge_form.find_all('input'):
+            if tag.get('name') is None:
+                continue
+
+            payload[tag.get('name')] = tag.get('value')
+
+        if response_page.find('input', {'name': 'TrustDevice'}) is not None:
+            payload['TrustDevice'] = 'on'
+
+        payload['Pin'] = sms_token
+
+        try:
+            del payload['SendMethod']
+        except KeyError:
+            pass
 
         # Submit IPP (SMS code)
         return self.post(challenge_url, data=payload)
@@ -605,7 +584,7 @@ class Google:
         response = None
         while retry:
             try:
-                response = self.post(await_url, json=await_body)
+                response = self.post(await_url, json_data=await_body)
                 retry = False
             except requests.exceptions.HTTPError as ex:
 
@@ -826,102 +805,49 @@ class Google:
 
     def handle_selectchallenge(self, sess):
         response_page = BeautifulSoup(sess.text, 'html.parser')
-        # Known mfa methods, 5 is disabled till its implemented
-        auth_methods = {
-            2: 'TOTP (Google Authenticator)',
-            3: 'SMS',
-            4: 'OOTP (Google Prompt)'
-            # 5: 'OOTP (Google App Offline Security Code)'
-        }
-
-        unavailable_challenge_ids = [
-            int(i.attrs.get('data-unavailable'))
-            for i in response_page.find_all(
-                lambda tag: tag.name == 'form' and 'data-unavailable' in tag.attrs
-            )
-        ]
-
-        # ootp via google app offline code isn't implemented. make sure its not valid.
-        unavailable_challenge_ids.append(5)
 
-        challenge_ids = [
-            int(i.get('value'))
-            for i in response_page.find_all('input', {'name': 'challengeId'})
-            if int(i.get('value')) not in unavailable_challenge_ids
-        ]
-
-        challenge_ids.sort()
-
-        auth_methods = {
-            k: auth_methods[k]
-            for k in challenge_ids
-            if k in auth_methods and k not in unavailable_challenge_ids
-        }
+        challenges = []
+        for i in response_page.select('form[data-challengeentry]'):
+            action = i.attrs.get("action")
+
+            if "challenge/totp/" in action:
+                challenges.append(['TOTP (Google Authenticator)', i.attrs.get("data-challengeentry")])
+            elif "challenge/ipp/" in action:
+                challenges.append(['SMS', i.attrs.get("data-challengeentry")])
+            elif "challenge/iap/" in action:
+                challenges.append(['SMS other phone', i.attrs.get("data-challengeentry")])
+            elif "challenge/sk/" in action:
+                challenges.append(['YubiKey', i.attrs.get("data-challengeentry")])
+            elif "challenge/az/" in action:
+                challenges.append(['Google Prompt', i.attrs.get("data-challengeentry")])
 
         print('Choose MFA method from available:')
-        print('\n'.join(
-            '{}: {}'.format(*i) for i in list(auth_methods.items())))
+        for i, mfa in enumerate(challenges, start=1):
+            print("{}: {}".format(i, mfa[0]))
 
-        selected_challenge = input("Enter MFA choice number ({}): ".format(
-            challenge_ids[-1:][0])) or None
+        selected_challenge = input("Enter MFA choice number (1): ") or None
 
-        if selected_challenge is not None and int(selected_challenge) in challenge_ids:
-            challenge_id = int(selected_challenge)
+        if selected_challenge is not None and int(selected_challenge) <= len(challenges):
+            selected_challenge = int(selected_challenge) - 1
         else:
-            # use the highest index as that will default to prompt, then sms, then totp, etc.
-            challenge_id = challenge_ids[-1:][0]
+            selected_challenge = 0
 
-        print("MFA Type Chosen: {}".format(auth_methods[challenge_id]))
+        challenge_id = challenges[selected_challenge][1]
+        print("MFA Type Chosen: {}".format(challenges[selected_challenge][0]))
 
         # We need the specific form of the challenge chosen
         challenge_form = response_page.find(
             'form', {'data-challengeentry': challenge_id})
 
-        payload = {
-            'challengeId':
-            challenge_id,
-            'challengeType':
-            challenge_form.find('input', {
-                'name': 'challengeType'
-            }).get('value'),
-            'continue':
-            challenge_form.find('input', {
-                'name': 'continue'
-            }).get('value'),
-            'scc':
-            challenge_form.find('input', {
-                'name': 'scc'
-            }).get('value'),
-            'sarp':
-            challenge_form.find('input', {
-                'name': 'sarp'
-            }).get('value'),
-            'checkedDomains':
-            challenge_form.find('input', {
-                'name': 'checkedDomains'
-            }).get('value'),
-            'pstMsg':
-            challenge_form.find('input', {
-                'name': 'pstMsg'
-            }).get('value'),
-            'TL':
-            challenge_form.find('input', {
-                'name': 'TL'
-            }).get('value'),
-            'gxf':
-            challenge_form.find('input', {
-                'name': 'gxf'
-            }).get('value'),
-            'subAction':
-            challenge_form.find('input', {
-                'name': 'subAction'
-            }).get('value'),
-        }
-        if challenge_form.find('input', {'name': 'SendMethod'}) is not None:
-            payload['SendMethod'] = challenge_form.find(
-                'input', {
-                    'name': 'SendMethod'
-                }).get('value')
+        payload = {}
+        for tag in challenge_form.find_all('input'):
+            if tag.get('name') is None:
+                continue
+
+            payload[tag.get('name')] = tag.get('value')
+
+        if response_page.find('input', {'name': 'TrustDevice'}) is not None:
+            payload['TrustDevice'] = 'on'
 
         # POST to google with the chosen challenge
         return self.post(


Test patch:
diff --git a/aws_google_auth/tests/test_args_parser.py b/aws_google_auth/tests/test_args_parser.py
index c934600..ad55116 100644
--- a/aws_google_auth/tests/test_args_parser.py
+++ b/aws_google_auth/tests/test_args_parser.py
@@ -36,10 +36,11 @@ class TestPythonFailOnVersion(unittest.TestCase):
         self.assertEqual(parser.account, None)
 
         self.assertFalse(parser.save_failure_html)
+        self.assertFalse(parser.save_saml_flow)
 
         # Assert the size of the parameter so that new parameters trigger a review of this function
         # and the appropriate defaults are added here to track backwards compatibility in the future.
-        self.assertEqual(len(vars(parser)), 20)
+        self.assertEqual(len(vars(parser)), 21)
 
     def test_username(self):
 
diff --git a/aws_google_auth/tests/test_init.py b/aws_google_auth/tests/test_init.py
index 8efcdca..fbdfcad 100644
--- a/aws_google_auth/tests/test_init.py
+++ b/aws_google_auth/tests/test_init.py
@@ -55,6 +55,7 @@ class TestInit(unittest.TestCase):
                                          resolve_aliases=False,
                                          role_arn=None,
                                          save_failure_html=False,
+                                         save_saml_flow=False,
                                          saml_cache=True,
                                          saml_assertion=None,
                                          sp_id=None,
@@ -78,6 +79,7 @@ class TestInit(unittest.TestCase):
                                          resolve_aliases=False,
                                          role_arn=None,
                                          save_failure_html=False,
+                                         save_saml_flow=False,
                                          saml_cache=True,
                                          saml_assertion=None,
                                          sp_id=None,