Repository: amundsen-io/amundsen

Issue:
Bug Report: Glue search_tables with Filters and result tables more than 100 items
<!--- Provide a general summary of the issue in the Title above -->
<!--- Look through existing open and closed issues to see if someone has reported the issue before -->

There is a bug while applying Filters for the database which contains more than 100 items. Since glue returns 100 items per page and to move to the next page we need to specify `NextToken`.
I have 138 tables, which means I will be iterating 2 times over the result.

The filter:
```python
{
        'Key': 'DatabaseName',
        'Value': glue_database_name
}
```

Every time I run the code I get different results: the length of the list is always the same - 138. However, the length of the set is always different. It ranges from 1 to 30.
I run my check over 10 times.

I took look at the documentation and found a proper parameter `MaxResults` for further checking. Since I know precisely desired table count, I put it as 150 and the issue has totally gone.

## Expected Behavior
Get the exact same result for filtered tables.

## Current Behavior
Query result from [`self._glue.search_tables(**kwargs)`](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/glue_extractor.py#L78) contains duplicates

## Possible Solution
I'm not sure, but I think for the next (second) iteration (page, which contains up to 100 items) we are using a new `NextToken` with previous filters. Maybe the problem lies here.

## Steps to Reproduce
1. Have more than 100 glue tables in a single DB in AWS
2. Query it using the abovementioned `DatabaseName` filter
3. Observe duplicates in the list

## Hot-fix
1. Add `MaxResults` to [`kwargs`](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/glue_extractor.py#L80) that is more than your actual size of overall tables
2. Observe a proper behavior

## Context
Q: How has this issue affected you?
A: It affects our production system

## Your Environment
```
amundsen-databuilder==4.5.3
amundsen-gremlin==0.0.9
Flask==1.1.4
gremlinpython==3.4.9
requests-aws4auth==1.1.1
typing-extensions==3.10.0
overrides==6.1.0
```


Discussion:

Resolving patch:
diff --git a/databuilder/databuilder/extractor/glue_extractor.py b/databuilder/databuilder/extractor/glue_extractor.py
--- a/databuilder/databuilder/extractor/glue_extractor.py
+++ b/databuilder/databuilder/extractor/glue_extractor.py
@@ -19,12 +19,14 @@ class GlueExtractor(Extractor):
 
     CLUSTER_KEY = 'cluster'
     FILTER_KEY = 'filters'
-    DEFAULT_CONFIG = ConfigFactory.from_dict({CLUSTER_KEY: 'gold', FILTER_KEY: None})
+    MAX_RESULTS_KEY = 'max_results'
+    DEFAULT_CONFIG = ConfigFactory.from_dict({CLUSTER_KEY: 'gold', FILTER_KEY: None, MAX_RESULTS_KEY: 500})
 
     def init(self, conf: ConfigTree) -> None:
         conf = conf.with_fallback(GlueExtractor.DEFAULT_CONFIG)
         self._cluster = conf.get_string(GlueExtractor.CLUSTER_KEY)
         self._filters = conf.get(GlueExtractor.FILTER_KEY)
+        self._max_results = conf.get(GlueExtractor.MAX_RESULTS_KEY)
         self._glue = boto3.client('glue')
         self._extract_iter: Union[None, Iterator] = None
 
@@ -80,6 +82,7 @@ def _search_tables(self) -> List[Dict[str, Any]]:
         kwargs = {}
         if self._filters is not None:
             kwargs['Filters'] = self._filters
+            kwargs['MaxResults'] = self._max_results
         data = self._glue.search_tables(**kwargs)
         tables += data['TableList']
         while 'NextToken' in data:


Test patch: