Repository: ansible-collections/amazon.aws

Issue:
amazon.aws.aws_secret lookup with bypath=true returns only upto 10 values
### Summary

I found an issue when using ansible lookup plugin. In my ansible playbook ( Ansible 2.10) I'm using lookup plugin to get all secrets from AWS Secrets Manager with the bypath attribute.

```
vars: 
#which environment: dev, test,uat, prd 
my_env: dev 
aws_secret_path: mypath/{{my_env}}

name: "get all secrets from AWS Secrets Manager" 
  set_fact: secret_value: "{{ lookup('amazon.aws.aws_secret', '{{aws_secret_path}}', on_missing='skip', bypath='true', region='eu-west-1' )}}"
```

It is working fine however I realized that it returns only up to 10 elements.  
Is there a way to return all elements or use pagination ? Have you faced similar issue with lookup plugin ?


### Issue Type

Bug Report

### Component Name

amazon.aws.aws_secret

### Ansible Version

```
 ansible --version
ansible 2.10.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /venv/awx-venv/lib64/python3.6/site-packages/ansible
  executable location = /venv/awx-venv/bin/ansible
  python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
```

### Collection Versions

```
Collection        Version
----------------- -------
amazon.aws        1.5.0  
community.aws     1.4.0  
community.general 2.3.0 
```

### AWS SDK versions

```
(awx-venv) bash-4.4$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /venv/awx-venv/lib/python3.6/site-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.17.46
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /venv/awx-venv/lib/python3.6/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.20.46
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /venv/awx-venv/lib/python3.6/site-packages
Requires: python-dateutil, jmespath, urllib3
Required-by: s3transfer, boto3
```

### Configuration

```
$ ansible-config dump --only-changed

```


### OS / Environment

_No response_

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
vars: 
#which environment: dev, test,uat, prd 
my_env: dev 
aws_secret_path: mypath/{{my_env}}

name: "get all secrets from AWS Secrets Manager" 
  set_fact: secret_value: "{{ lookup('amazon.aws.aws_secret', '{{aws_secret_path}}', on_missing='skip', bypath='true', region='eu-west-1' )}}"
```


### Expected Results

Lookup plugin should return all secrets from AWS Secret Manager

### Actual Results

```console (paste below)
It returns only up tp 10 results
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
None

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
As sugested by mdaniel on Stackoverflow It seems you are  ignoring the advice to honor NextToken - 
[as described here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.list_secrets) Kindly please implement it as it doesn't allow to fetch more than 10 records from AWS Secrets Manager
hi! any update on this issue ? Would good if that could be fixed. 
Thanks!
Greg
> hi! any update on this issue ? Would good if that could be fixed. Thanks! Greg

Hi Greg, can you please try with this patch https://github.com/ansible-collections/amazon.aws/pull/591? 

Hi Mandar,

Thank you for letting me know. Would you be so kind and share with me some info on how can I test that ? I'm not so familiar with ansible galaxy and I'm not sure how can I obtain the release containing this patch and how can I use it.

Kindly please let me know.

Thank you and best regards,
Greg




From: Mandar Kulkarni ***@***.***>
Sent: Monday, January 17, 2022 10:56 AM
To: ansible-collections/amazon.aws ***@***.***>
Cc: Grzesiuk, Grzegorz /PL ***@***.***>; Author ***@***.***>
Subject: Re: [ansible-collections/amazon.aws] amazon.aws.aws_secret lookup with bypath=true returns only upto 10 values (#472)


hi! any update on this issue ? Would good if that could be fixed. Thanks! Greg

Hi Greg, can you please try with this patch #591<https://github.com/ansible-collections/amazon.aws/pull/591?>?

-
Reply to this email directly, view it on GitHub<https://github.com/ansible-collections/amazon.aws/issues/472#issuecomment-1014335280>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AU3XZPEITG2IWOJZCLVVSRLUWPRSFANCNFSM5CZECBKQ>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

The information contained in the e-mail or attached document is CONFIDENTIAL and is intended only for the use of the individual to whom it is addressed. If you are not the intended recipient or employee responsible to deliver to the intended recipient, you are hereby notified that any dissemination, distribution or copying of the communication is strictly prohibited. If you have received the e-mail in error, please immediately notify us and return the original message to us at the address above or destroy it

________________________________

Zawarte w niniejszej wiadomo?ci lub za??czonym dokumencie informacje s? POUFNE i przeznaczone tylko dla oznaczonego adresata. Rozpowszechnianie, ujawnianie i kopiowanie tych informacji jest zabronione. Je?eli niniejsza wiadomo?? trafi?a do Pa?stwa przez pomy?k?, bardzo prosimy o powiadomienie nas o tym fakcie i odes?anie dokumentu lub natychmiastowe jego zniszczenie

________________________________

Sanofi-Aventis Sp. z o.o. z siedzib? w Warszawie, przy ul. Bonifraterskiej 17 , 00-203 Warszawa, wpisana do rejestru przedsi?biorc?w Krajowego Rejestru S?dowego prowadzonego przez S?d Rejonowy dla miasta sto?ecznego Warszawy , XII Wydzia? Gospodarczy Krajowego Rejestru S?dowego, pod numerem KRS 0000036286, NIP 813-01-40-525, kapita? zak?adowy w wysoko?ci 50 738 386,94z?.


Resolving patch:
diff --git a/plugins/lookup/aws_secret.py b/plugins/lookup/aws_secret.py
--- a/plugins/lookup/aws_secret.py
+++ b/plugins/lookup/aws_secret.py
@@ -218,16 +218,18 @@ def run(self, terms, variables=None, boto_profile=None, aws_profile=None,
             secrets = {}
             for term in terms:
                 try:
-                    response = client.list_secrets(Filters=[{'Key': 'name', 'Values': [term]}])
+                    paginator = client.get_paginator('list_secrets')
+                    paginator_response = paginator.paginate(
+                        Filters=[{'Key': 'name', 'Values': [term]}])
+                    for object in paginator_response:
+                        if 'SecretList' in object:
+                            for secret_obj in object['SecretList']:
+                                secrets.update({secret_obj['Name']: self.get_secret_value(
+                                    secret_obj['Name'], client, on_missing=missing, on_denied=denied)})
+                    secrets = [secrets]
 
-                    if 'SecretList' in response:
-                        for secret in response['SecretList']:
-                            secrets.update({secret['Name']: self.get_secret_value(secret['Name'], client,
-                                                                                  on_missing=missing,
-                                                                                  on_denied=denied)})
                 except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
                     raise AnsibleError("Failed to retrieve secret: %s" % to_native(e))
-            secrets = [secrets]
         else:
             secrets = []
             for term in terms:


Test patch:
diff --git a/tests/unit/plugins/lookup/test_aws_secret.py b/tests/unit/plugins/lookup/test_aws_secret.py
--- a/tests/unit/plugins/lookup/test_aws_secret.py
+++ b/tests/unit/plugins/lookup/test_aws_secret.py
@@ -163,7 +163,7 @@ def test_nested_lookup_variable(mocker, dummy_credentials):
                                            aws_secret_access_key="notasecret", aws_session_token=None)
 
 
-def test_path_lookup_variable(mocker, dummy_credentials):
+def test_path_lookup_variable(mocker, dummy_credentials, record_property):
     lookup = aws_secret.LookupModule()
     lookup._load_name = "aws_secret"
 
@@ -205,14 +205,84 @@ def test_path_lookup_variable(mocker, dummy_credentials):
     ]
 
     boto3_client_double = boto3_double.Session.return_value.client
+    boto3_client_get_paginator_double = boto3_double.Session.return_value.client.return_value.get_paginator
+    boto3_paginate_double = boto3_client_get_paginator_double.return_value.paginate
+    boto3_paginate_double.return_value = [path_list_secrets_success_response]
 
     mocker.patch.object(boto3, 'session', boto3_double)
     dummy_credentials["bypath"] = 'true'
     dummy_credentials["boto_profile"] = 'test'
     dummy_credentials["aws_profile"] = 'test'
     retval = lookup.run(["/testpath"], {}, **dummy_credentials)
+    boto3_paginate_double.assert_called_once()
+    boto3_client_get_paginator_double.assert_called_once()
+    boto3_client_get_paginator_double.assert_called_once_with('list_secrets')
+
+    record_property('KEY', retval)
     assert (retval[0]["/testpath/won"] == "simple_value_won")
     assert (retval[0]["/testpath/too"] == "simple_value_too")
     boto3_client_double.assert_called_with('secretsmanager', 'eu-west-1', aws_access_key_id='notakey',
                                            aws_secret_access_key="notasecret", aws_session_token=None)
-    list_secrets_fn.assert_called_with(Filters=[{'Key': 'name', 'Values': ['/testpath']}])
+    boto3_paginate_double.assert_called_with(Filters=[{'Key': 'name', 'Values': ['/testpath']}])
+
+
+def test_path_lookup_variable_paginated(mocker, dummy_credentials, record_property):
+    lookup = aws_secret.LookupModule()
+    lookup._load_name = "aws_secret"
+
+    def secret(value):
+        return {"Name": f"/testpath_paginated/{value}"}
+
+    def get_secret_list(value):
+        return [secret(f"{value}{i}") for i in range(0, 6)]
+    path_list_secrets_paginated_success_response = {
+        'SecretList': [
+            item for pair in zip(get_secret_list("too"), get_secret_list("won")) for item in pair
+        ],
+        'ResponseMetadata': {
+            'RequestId': '21099462-597c-490a-800f-8b7a41e5151c',
+            'HTTPStatusCode': 200,
+            'HTTPHeaders': {
+                'date': 'Thu, 04 Apr 2019 10:43:12 GMT',
+                'content-type': 'application/x-amz-json-1.1',
+                'content-length': '252',
+                'connection': 'keep-alive',
+                'x-amzn-requestid': '21099462-597c-490a-800f-8b7a41e5151c'
+            },
+            'RetryAttempts': 0
+        }
+    }
+    boto3_double = mocker.MagicMock()
+    list_secrets_fn = boto3_double.Session.return_value.client.return_value.list_secrets
+    list_secrets_fn.return_value = path_list_secrets_paginated_success_response
+    get_secret_value_fn = boto3_double.Session.return_value.client.return_value.get_secret_value
+
+    def secret_string(val):
+        path = copy(simple_variable_success_response)
+        path["SecretString"] = f"simple_value_{val}"
+        return path
+
+    def _get_secret_list(value):
+        return [secret_string(f"{value}{i}") for i in range(0, 6)]
+    get_secret_value_fn.side_effect = [
+        item for pair in zip(_get_secret_list("too"), _get_secret_list("won")) for item in pair
+    ]
+    boto3_client_double = boto3_double.Session.return_value.client
+    boto3_client_get_paginator_double = boto3_double.Session.return_value.client.return_value.get_paginator
+    boto3_paginate_double = boto3_client_get_paginator_double.return_value.paginate
+    boto3_paginate_double.return_value = [path_list_secrets_paginated_success_response]
+    mocker.patch.object(boto3, 'session', boto3_double)
+    dummy_credentials["bypath"] = 'true'
+    dummy_credentials["boto_profile"] = 'test'
+    dummy_credentials["aws_profile"] = 'test'
+    retval = lookup.run(["/testpath_paginated"], {}, **dummy_credentials)
+    boto3_paginate_double.assert_called_once()
+    boto3_client_get_paginator_double.assert_called_once()
+    boto3_client_get_paginator_double.assert_called_once_with('list_secrets')
+    record_property('KEY', retval)
+    assert (retval[0]["/testpath_paginated/won0"] == "simple_value_won0")
+    assert (retval[0]["/testpath_paginated/too0"] == "simple_value_too0")
+    assert (len(retval[0]) == 12)
+    boto3_client_double.assert_called_with('secretsmanager', 'eu-west-1', aws_access_key_id='notakey',
+                                           aws_secret_access_key="notasecret", aws_session_token=None)
+    boto3_paginate_double.assert_called_with(Filters=[{'Key': 'name', 'Values': ['/testpath_paginated']}])