Repository: aws/aws-sdk-pandas

Issue:
wr.catalog.sanitize_dataframe_columns_names does not sanitize enough
# Issue
`wr.catalog.sanitize_dataframe_columns_names` does not correctly sanitize CammelCase to snake_case. It also allows duplicate columns which would be problematic in Athena.

# Example
```
import pandas as pd
import awswrangler as wr
print(wr.__version__)
# 2.13.0
df = pd.DataFrame({'SomeLongStringWithMultipleWords': [1, 2, 3], 'sinceEpoch': [4, 5, 6], 'SinceEpoch': [7, 8, 9], 'since_epoch': [7, 8, 9]})
df_clean = wr.catalog.sanitize_dataframe_columns_names(df)
print(df_clean)
# somelongstringwithmultiplewords  sinceepoch  sinceepoch  since_epoch
# 0                                1           4           7            7
# 1                                2           5           8            8
# 2                                3           6           9            9

print(df_clean["sinceepoch"])
#    sinceepoch  sinceepoch
# 0           4           7
# 1           5           8
# 2           6           9
```

## First issue
Based on the [documentation](https://aws-data-wrangler.readthedocs.io/en/stable/stubs/awswrangler.catalog.sanitize_dataframe_columns_names.html) for this function, I would have expected each column name to be snake_case.


## Second issue
This is more of a judgement call and I could see reasons for not adding it, however, to live up to the function name, if duplicate columns would either throw an error or have a suffix added to them. I'd appreciate it if this was at least optional. 





Discussion:
I actually wrote (copied from StackOverflow) two functions to do this before I discovered your excellent library.

```
def camel_to_snake(name):
    """
    Converts camelCase to snake_case.
    """
    s1 = re.sub("(.)([A-Z][a-z]+)", r"\1_\2", name)
    return re.sub("([a-z0-9])([A-Z])", r"\1_\2", s1).lower()
```

https://stackoverflow.com/questions/1175208/elegant-python-function-to-convert-camelcase-to-snake-case

And 

```
def check_key_conflicts(new_key, d: dict):
    """
    Checks if the new key is already in the dict.
    """
    if new_key in d.keys():
        new_key = new_key + "_conflict" # Could maybe be an argument too.
        return check_key_conflicts(new_key, d) # Do this iteratively to avoid the next column being named `conflict`
    return new_key
```
Also, my apologies, I didn't see the `drop_duplicate_columns` function which will help with this. I might still suggest that you implement an option to add a suffix/number to columns that have been cleaned.

I can put in a PR to fix the snake case issue. I also have code to add a suffix to the duplicate columns but, I don't know if you want to go in that direciton.
Hi @kailukowiak, thanks for raising this. The Wrangler sanitize methods are meant to replicate the sanitize functionality available in the AWS Glue service. This is because the Glue catalog is used as a Hive metastore for a number of methods in the library. In Glue, a string such as `sinceEpoch` would be transformed to `sinceepoch`. You can test this by creating a Glue database, table or column in the console.

However, I do agree with you that the documentation is misleading and should not refer to snake case conversion. I will make sure to update that.

Regarding the second issue, the `drop_duplicate_columns` flag is indeed intended for this corner case. In general, we try to avoid modifying pandas dataframes metadata and stick to whatever is already allowed in the library (i.e. in this case, duplicate columns are allowed in Pandas). That being said, I would be in favor of adding a warning statement when [duplicate](https://towardsdatascience.com/a-few-times-i-managed-to-broke-pandas-d3604d43708c) columns are detected. A PR for this feature would be welcome if you are interested
As a side note, there is a private [check_duplicate_columns](https://github.com/awslabs/aws-data-wrangler/blob/main/awswrangler/_utils.py#L309) method that can be leveraged in the above PR but feel free to improve on it
Interesting point. Aligning with Glue makes sense and while I prefer snake for legibility you're right that it should align with Glue.

Would you consider a `modify_duplicate_columns` function? I ask because renaming columns is less drastic of a step than deleting them.

I also think this may be more common than the edge cases, especially when reading JSON data into a DF. For example, I have two versions of an app for iOS and Android where one reports `"userId":123` and the other `"userid":456`. Pandas reads them as different columns but then `wr.catalog.sanitize_dataframe_columns_names` would make them identical. A warning here would be ok as the best practice might be to coalesce the columns but at least for my use case, I'm going to rename them instead.

I guess I'm trying to say that I see `modify_duplicate_columns` being at least as useful as `drop_duplicate_columns`. However, if you want to keep the library a bit slimmer I get that and can just write a warning for when there are duplicates after calling `wr.catalog.sanitize_dataframe_columns_names`
That is fair enough. Yes, please go ahead an add a flag if you can. I would suggest `rename_duplicate_columns` to be clearer. If neither the `rename` nor `drop` flags are used and there are duplicate cols, then a warning should be raised. How is that?

Resolving patch:
diff --git a/awswrangler/catalog/__init__.py b/awswrangler/catalog/__init__.py
--- a/awswrangler/catalog/__init__.py
+++ b/awswrangler/catalog/__init__.py
@@ -43,6 +43,7 @@
     does_table_exist,
     drop_duplicated_columns,
     extract_athena_types,
+    rename_duplicated_columns,
     sanitize_column_name,
     sanitize_dataframe_columns_names,
     sanitize_table_name,
@@ -57,6 +58,7 @@
     "delete_column",
     "drop_duplicated_columns",
     "extract_athena_types",
+    "rename_duplicated_columns",
     "sanitize_column_name",
     "sanitize_dataframe_columns_names",
     "sanitize_table_name",
diff --git a/awswrangler/catalog/_utils.py b/awswrangler/catalog/_utils.py
--- a/awswrangler/catalog/_utils.py
+++ b/awswrangler/catalog/_utils.py
@@ -2,6 +2,7 @@
 import logging
 import re
 import unicodedata
+import warnings
 from typing import Any, Dict, List, Optional, Tuple
 
 import boto3
@@ -124,8 +125,52 @@ def sanitize_column_name(column: str) -> str:
     return _sanitize_name(name=column)
 
 
-def sanitize_dataframe_columns_names(df: pd.DataFrame) -> pd.DataFrame:
-    """Normalize all columns names to be compatible with Amazon Athena and the AWS Glue Catalog.
+def rename_duplicated_columns(df: pd.DataFrame) -> pd.DataFrame:
+    """Append an incremental number to duplicate column names to conform with Amazon Athena.
+
+    Note
+    ----
+    This transformation will run `inplace` and will make changes to the original DataFrame.
+
+    Note
+    ----
+    Also handles potential new column duplicate conflicts by appending an additional `_n`.
+
+    Parameters
+    ----------
+    df : pandas.DataFrame
+        Original Pandas DataFrame.
+
+    Returns
+    -------
+    pandas.DataFrame
+        DataFrame with duplicated column names renamed.
+
+    Examples
+    --------
+    >>> df = pd.DataFrame({'a': [1, 2], 'b': [3, 4], 'c': [4, 6]})
+    >>> df.columns = ['a', 'a', 'a_1']
+    >>> wr.catalog.rename_duplicated_columns(df=df)
+    a	a_1	a_1_1
+    1	3	4
+    2	4	6
+    """
+    names = df.columns
+    set_names = set(names)
+    if len(names) == len(set_names):
+        return df
+    d = {key: [name + f"_{i}" if i > 0 else name for i, name in enumerate(names[names == key])] for key in set_names}
+    df.rename(columns=lambda c: d[c].pop(0), inplace=True)
+    while df.columns.duplicated().any():
+        # Catches edge cases where pd.DataFrame({"A": [1, 2], "a": [3, 4], "a_1": [5, 6]})
+        df = rename_duplicated_columns(df)
+    return df
+
+
+def sanitize_dataframe_columns_names(
+    df: pd.DataFrame, handle_duplicate_columns: Optional[str] = "warn"
+) -> pd.DataFrame:
+    """Normalize all columns names to be compatible with Amazon Athena.
 
     https://docs.aws.amazon.com/athena/latest/ug/tables-databases-columns-names.html
 
@@ -142,6 +187,11 @@ def sanitize_dataframe_columns_names(df: pd.DataFrame) -> pd.DataFrame:
     ----------
     df : pandas.DataFrame
         Original Pandas DataFrame.
+    handle_duplicate_columns : str, optional
+        How to handle duplicate columns. Can be "warn" or "drop" or "rename".
+        "drop" will drop all but the first duplicated column.
+        "rename" will rename all duplicated columns with an incremental number.
+        Defaults to "warn".
 
     Returns
     -------
@@ -151,11 +201,29 @@ def sanitize_dataframe_columns_names(df: pd.DataFrame) -> pd.DataFrame:
     Examples
     --------
     >>> import awswrangler as wr
-    >>> df_normalized = wr.catalog.sanitize_dataframe_columns_names(df=pd.DataFrame({'A': [1, 2]}))
+    >>> df_normalized = wr.catalog.sanitize_dataframe_columns_names(df=pd.DataFrame({"A": [1, 2]}))
+    >>> df_normalized_drop = wr.catalog.sanitize_dataframe_columns_names(
+            df=pd.DataFrame({"A": [1, 2], "a": [3, 4]}), handle_duplicate_columns="drop"
+        )
+    >>> df_normalized_rename = wr.catalog.sanitize_dataframe_columns_names(
+            df=pd.DataFrame({"A": [1, 2], "a": [3, 4], "a_1": [4, 6]}), handle_duplicate_columns="rename"
+        )
 
     """
     df.columns = [sanitize_column_name(x) for x in df.columns]
     df.index.names = [None if x is None else sanitize_column_name(x) for x in df.index.names]
+    if df.columns.duplicated().any():  # type: ignore
+        if handle_duplicate_columns == "warn":
+            warnings.warn(
+                "Duplicate columns were detected, consider using `handle_duplicate_columns='[drop|rename]'`",
+                UserWarning,
+            )
+        elif handle_duplicate_columns == "drop":
+            df = drop_duplicated_columns(df)
+        elif handle_duplicate_columns == "rename":
+            df = rename_duplicated_columns(df)
+        else:
+            raise ValueError("handle_duplicate_columns must be one of ['warn', 'drop', 'rename']")
     return df
 
 


Test patch:
diff --git a/tests/test_athena.py b/tests/test_athena.py
--- a/tests/test_athena.py
+++ b/tests/test_athena.py
@@ -246,6 +246,19 @@ def test_athena_read_list(glue_database):
         wr.athena.read_sql_query(sql="SELECT ARRAY[1, 2, 3]", database=glue_database, ctas_approach=False)
 
 
+def test_sanitize_dataframe_column_names():
+    with pytest.warns(UserWarning, match=r"Duplicate*"):
+        test_df = pd.DataFrame({"a": [1, 2], "b": [3, 4]})
+        test_df.columns = ["a", "a"]
+        assert wr.catalog.sanitize_dataframe_columns_names(df=pd.DataFrame({"A": [1, 2], "a": [3, 4]})).equals(test_df)
+    assert wr.catalog.sanitize_dataframe_columns_names(
+        df=pd.DataFrame({"A": [1, 2], "a": [3, 4]}), handle_duplicate_columns="drop"
+    ).equals(pd.DataFrame({"a": [1, 2]}))
+    assert wr.catalog.sanitize_dataframe_columns_names(
+        df=pd.DataFrame({"A": [1, 2], "a": [3, 4], "a_1": [5, 6]}), handle_duplicate_columns="rename"
+    ).equals(pd.DataFrame({"a": [1, 2], "a_1": [3, 4], "a_1_1": [5, 6]}))
+
+
 def test_sanitize_names():
     assert wr.catalog.sanitize_column_name("CamelCase") == "camelcase"
     assert wr.catalog.sanitize_column_name("CamelCase2") == "camelcase2"