Repository: ansible-collections/amazon.aws

Issue:
amazon.aws.aws_secret lookup with bypath=true returns only upto 10 values
### Summary

I found an issue when using ansible lookup plugin. In my ansible playbook ( Ansible 2.10) I'm using lookup plugin to get all secrets from AWS Secrets Manager with the bypath attribute.

```
vars: 
#which environment: dev, test,uat, prd 
my_env: dev 
aws_secret_path: mypath/{{my_env}}

name: "get all secrets from AWS Secrets Manager" 
  set_fact: secret_value: "{{ lookup('amazon.aws.aws_secret', '{{aws_secret_path}}', on_missing='skip', bypath='true', region='eu-west-1' )}}"
```

It is working fine however I realized that it returns only up to 10 elements.  
Is there a way to return all elements or use pagination ? Have you faced similar issue with lookup plugin ?


### Issue Type

Bug Report

### Component Name

amazon.aws.aws_secret

### Ansible Version

```
 ansible --version
ansible 2.10.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /venv/awx-venv/lib64/python3.6/site-packages/ansible
  executable location = /venv/awx-venv/bin/ansible
  python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
```

### Collection Versions

```
Collection        Version
----------------- -------
amazon.aws        1.5.0  
community.aws     1.4.0  
community.general 2.3.0 
```

### AWS SDK versions

```
(awx-venv) bash-4.4$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /venv/awx-venv/lib/python3.6/site-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.17.46
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /venv/awx-venv/lib/python3.6/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.20.46
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /venv/awx-venv/lib/python3.6/site-packages
Requires: python-dateutil, jmespath, urllib3
Required-by: s3transfer, boto3
```

### Configuration

```
$ ansible-config dump --only-changed

```


### OS / Environment

_No response_

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
vars: 
#which environment: dev, test,uat, prd 
my_env: dev 
aws_secret_path: mypath/{{my_env}}

name: "get all secrets from AWS Secrets Manager" 
  set_fact: secret_value: "{{ lookup('amazon.aws.aws_secret', '{{aws_secret_path}}', on_missing='skip', bypath='true', region='eu-west-1' )}}"
```


### Expected Results

Lookup plugin should return all secrets from AWS Secret Manager

### Actual Results

```console (paste below)
It returns only up tp 10 results
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
None

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
As sugested by mdaniel on Stackoverflow It seems you are  ignoring the advice to honor NextToken - 
[as described here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.list_secrets) Kindly please implement it as it doesn't allow to fetch more than 10 records from AWS Secrets Manager
hi! any update on this issue ? Would good if that could be fixed. 
Thanks!
Greg

Resolving patch:
diff --git a/plugins/lookup/aws_secret.py b/plugins/lookup/aws_secret.py
--- a/plugins/lookup/aws_secret.py
+++ b/plugins/lookup/aws_secret.py
@@ -218,16 +218,18 @@ def run(self, terms, variables=None, boto_profile=None, aws_profile=None,
             secrets = {}
             for term in terms:
                 try:
-                    response = client.list_secrets(Filters=[{'Key': 'name', 'Values': [term]}])
+                    paginator = client.get_paginator('list_secrets')
+                    paginator_response = paginator.paginate(
+                        Filters=[{'Key': 'name', 'Values': [term]}])
+                    for object in paginator_response:
+                        if 'SecretList' in object:
+                            for secret_obj in object['SecretList']:
+                                secrets.update({secret_obj['Name']: self.get_secret_value(
+                                    secret_obj['Name'], client, on_missing=missing, on_denied=denied)})
+                    secrets = [secrets]
 
-                    if 'SecretList' in response:
-                        for secret in response['SecretList']:
-                            secrets.update({secret['Name']: self.get_secret_value(secret['Name'], client,
-                                                                                  on_missing=missing,
-                                                                                  on_denied=denied)})
                 except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
                     raise AnsibleError("Failed to retrieve secret: %s" % to_native(e))
-            secrets = [secrets]
         else:
             secrets = []
             for term in terms:


Test patch:
diff --git a/tests/unit/plugins/lookup/test_aws_secret.py b/tests/unit/plugins/lookup/test_aws_secret.py
--- a/tests/unit/plugins/lookup/test_aws_secret.py
+++ b/tests/unit/plugins/lookup/test_aws_secret.py
@@ -163,7 +163,7 @@ def test_nested_lookup_variable(mocker, dummy_credentials):
                                            aws_secret_access_key="notasecret", aws_session_token=None)
 
 
-def test_path_lookup_variable(mocker, dummy_credentials):
+def test_path_lookup_variable(mocker, dummy_credentials, record_property):
     lookup = aws_secret.LookupModule()
     lookup._load_name = "aws_secret"
 
@@ -205,14 +205,84 @@ def test_path_lookup_variable(mocker, dummy_credentials):
     ]
 
     boto3_client_double = boto3_double.Session.return_value.client
+    boto3_client_get_paginator_double = boto3_double.Session.return_value.client.return_value.get_paginator
+    boto3_paginate_double = boto3_client_get_paginator_double.return_value.paginate
+    boto3_paginate_double.return_value = [path_list_secrets_success_response]
 
     mocker.patch.object(boto3, 'session', boto3_double)
     dummy_credentials["bypath"] = 'true'
     dummy_credentials["boto_profile"] = 'test'
     dummy_credentials["aws_profile"] = 'test'
     retval = lookup.run(["/testpath"], {}, **dummy_credentials)
+    boto3_paginate_double.assert_called_once()
+    boto3_client_get_paginator_double.assert_called_once()
+    boto3_client_get_paginator_double.assert_called_once_with('list_secrets')
+
+    record_property('KEY', retval)
     assert (retval[0]["/testpath/won"] == "simple_value_won")
     assert (retval[0]["/testpath/too"] == "simple_value_too")
     boto3_client_double.assert_called_with('secretsmanager', 'eu-west-1', aws_access_key_id='notakey',
                                            aws_secret_access_key="notasecret", aws_session_token=None)
-    list_secrets_fn.assert_called_with(Filters=[{'Key': 'name', 'Values': ['/testpath']}])
+    boto3_paginate_double.assert_called_with(Filters=[{'Key': 'name', 'Values': ['/testpath']}])
+
+
+def test_path_lookup_variable_paginated(mocker, dummy_credentials, record_property):
+    lookup = aws_secret.LookupModule()
+    lookup._load_name = "aws_secret"
+
+    def secret(value):
+        return {"Name": f"/testpath_paginated/{value}"}
+
+    def get_secret_list(value):
+        return [secret(f"{value}{i}") for i in range(0, 6)]
+    path_list_secrets_paginated_success_response = {
+        'SecretList': [
+            item for pair in zip(get_secret_list("too"), get_secret_list("won")) for item in pair
+        ],
+        'ResponseMetadata': {
+            'RequestId': '21099462-597c-490a-800f-8b7a41e5151c',
+            'HTTPStatusCode': 200,
+            'HTTPHeaders': {
+                'date': 'Thu, 04 Apr 2019 10:43:12 GMT',
+                'content-type': 'application/x-amz-json-1.1',
+                'content-length': '252',
+                'connection': 'keep-alive',
+                'x-amzn-requestid': '21099462-597c-490a-800f-8b7a41e5151c'
+            },
+            'RetryAttempts': 0
+        }
+    }
+    boto3_double = mocker.MagicMock()
+    list_secrets_fn = boto3_double.Session.return_value.client.return_value.list_secrets
+    list_secrets_fn.return_value = path_list_secrets_paginated_success_response
+    get_secret_value_fn = boto3_double.Session.return_value.client.return_value.get_secret_value
+
+    def secret_string(val):
+        path = copy(simple_variable_success_response)
+        path["SecretString"] = f"simple_value_{val}"
+        return path
+
+    def _get_secret_list(value):
+        return [secret_string(f"{value}{i}") for i in range(0, 6)]
+    get_secret_value_fn.side_effect = [
+        item for pair in zip(_get_secret_list("too"), _get_secret_list("won")) for item in pair
+    ]
+    boto3_client_double = boto3_double.Session.return_value.client
+    boto3_client_get_paginator_double = boto3_double.Session.return_value.client.return_value.get_paginator
+    boto3_paginate_double = boto3_client_get_paginator_double.return_value.paginate
+    boto3_paginate_double.return_value = [path_list_secrets_paginated_success_response]
+    mocker.patch.object(boto3, 'session', boto3_double)
+    dummy_credentials["bypath"] = 'true'
+    dummy_credentials["boto_profile"] = 'test'
+    dummy_credentials["aws_profile"] = 'test'
+    retval = lookup.run(["/testpath_paginated"], {}, **dummy_credentials)
+    boto3_paginate_double.assert_called_once()
+    boto3_client_get_paginator_double.assert_called_once()
+    boto3_client_get_paginator_double.assert_called_once_with('list_secrets')
+    record_property('KEY', retval)
+    assert (retval[0]["/testpath_paginated/won0"] == "simple_value_won0")
+    assert (retval[0]["/testpath_paginated/too0"] == "simple_value_too0")
+    assert (len(retval[0]) == 12)
+    boto3_client_double.assert_called_with('secretsmanager', 'eu-west-1', aws_access_key_id='notakey',
+                                           aws_secret_access_key="notasecret", aws_session_token=None)
+    boto3_paginate_double.assert_called_with(Filters=[{'Key': 'name', 'Values': ['/testpath_paginated']}])