Repository: ansible-collections/amazon.aws

Issue:
The cloudwatchlogs_log_group_info module generates throttling exceptions
### Summary

When running the amazon.aws.cloudwatchlogs_log_group_info module on AWS accounts having more than (roughly) 50 log groups we get ThrottlingExceptions once every 20 calls or so. I noticed that both the describe and the list-tags boto calls in the cloudwatchlogs_log_group_info module have no throttling handling configured and use the default throttling handling policy, ie. none. 

### Issue Type

Bug Report

### Component Name

amazon.aws.cloudwatchlogs_log_group_info

### Ansible Version

```console (paste below)
ansible [core 2.13.13]
  config file = None
  configured module search path = ['/home/rundeck/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  ansible collection location = /home/rundeck/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.8.10 (default, Nov 22 2023, 10:22:35) [GCC 9.4.0]
  jinja version = 3.1.3
  libyaml = True

```


### Collection Versions

```console (paste below)
Collection            Version
--------------------- -------
amazon.aws            7.3.0
community.general     8.3.0
nutanix.ncp           1.7.0
servicenow.servicenow 1.0.6

# /usr/local/lib/python3.8/dist-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    3.5.0
ansible.netcommon             3.1.3
ansible.posix                 1.4.0
ansible.utils                 2.8.0
ansible.windows               1.12.0
arista.eos                    5.0.1
awx.awx                       21.10.0
azure.azcollection            1.14.0
check_point.mgmt              2.3.0
chocolatey.chocolatey         1.3.1
cisco.aci                     2.3.0
cisco.asa                     3.1.0
cisco.dnac                    6.6.1
cisco.intersight              1.0.22
cisco.ios                     3.3.2
cisco.iosxr                   3.3.1
cisco.ise                     2.5.9
cisco.meraki                  2.13.0
cisco.mso                     2.1.0
cisco.nso                     1.0.3
cisco.nxos                    3.2.0
cisco.ucs                     1.8.0
cloud.common                  2.1.2
cloudscale_ch.cloud           2.2.3
community.aws                 3.6.0
[...]
```


### AWS SDK versions

```console (paste below)
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.34.45
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /usr/local/lib/python3.8/dist-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.34.45
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /usr/local/lib/python3.8/dist-packages
Requires: python-dateutil, jmespath, urllib3
Required-by: s3transfer, boto3, awscli
```


### Configuration

```console (paste below)
(no Ansible configuration)
```


### OS / Environment

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"

### Steps to Reproduce

```yaml 
- name: "Role based get all log groups in {{ selected_region }}"
  amazon.aws.cloudwatchlogs_log_group_info:
    region: "{{ selected_region }}"
    aws_access_key: "{{ aws_access_key }}"
    aws_secret_key: "{{ aws_secret_key }}"
    aws_session_token: "{{ aws_session_token }}"
    log_group_name: "{{ log_group_prefix }}"
  register: log_groups
```


### Expected Results

Should get all log group information and tags without error. Optionally, allow us to add an option to skip getting the tags for each log group would allow engineers to work around this issue. 

### Actual Results

```console (paste below)
fatal: [127.0.0.1]: FAILED! => {"boto3_version": "1.34.45", "botocore_version": "1.34.45", "changed": false, "error": {"code": "ThrottlingException", "message": "Rate exceeded"}, "msg": "Unable to describe tags for log group /aws/codebuild/tankmaintenanceplanning-pipeline-tsa: An error occurred (ThrottlingException) when calling the ListTagsLogGroup operation (reached max retries: 4): Rate exceeded"
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
@b0tting the way we prevent throttling issues when making API calls is by implementing exponential backoff.
Could you please give it a try with #2019 and see if this is solving your issue?
Thanks
Ansible tasks run successfully. I don't have definite proof that this solves the issue, I don't see tasks taking more time than expected (ie. timed backoffs), but I expect this is the solution. Thank you!

Resolving patch:
diff --git a/plugins/modules/cloudwatchlogs_log_group_info.py b/plugins/modules/cloudwatchlogs_log_group_info.py
--- a/plugins/modules/cloudwatchlogs_log_group_info.py
+++ b/plugins/modules/cloudwatchlogs_log_group_info.py
@@ -82,6 +82,18 @@
 
 from ansible_collections.amazon.aws.plugins.module_utils.modules import AnsibleAWSModule
 from ansible_collections.amazon.aws.plugins.module_utils.botocore import is_boto3_error_code
+from ansible_collections.amazon.aws.plugins.module_utils.retries import AWSRetry
+
+
+@AWSRetry.exponential_backoff()
+def list_tags_log_group_with_backoff(client, log_group_name):
+    return client.list_tags_log_group(logGroupName=log_group_name)
+
+
+@AWSRetry.exponential_backoff()
+def describe_log_groups_with_backoff(client, **kwargs):
+    paginator = client.get_paginator("describe_log_groups")
+    return paginator.paginate(**kwargs).build_full_result()
 
 
 def describe_log_group(client, log_group_name, module):
@@ -89,15 +101,14 @@ def describe_log_group(client, log_group_name, module):
     if log_group_name:
         params["logGroupNamePrefix"] = log_group_name
     try:
-        paginator = client.get_paginator("describe_log_groups")
-        desc_log_group = paginator.paginate(**params).build_full_result()
+        desc_log_group = describe_log_groups_with_backoff(client, **params)
     except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
         module.fail_json_aws(e, msg=f"Unable to describe log group {log_group_name}")
 
     for log_group in desc_log_group["logGroups"]:
         log_group_name = log_group["logGroupName"]
         try:
-            tags = client.list_tags_log_group(logGroupName=log_group_name)
+            tags = list_tags_log_group_with_backoff(client, log_group_name)
         except is_boto3_error_code("AccessDeniedException"):
             tags = {}
             module.warn(f"Permission denied listing tags for log group {log_group_name}")


Test patch: