Repository: pyjanitor-devs/pyjanitor

Issue:
[ENH] Proposal: Enforce automagic column name conversion to string type in `clean_names`
UPDATE FROM MAINTAINERS: ANYBODY WHO IS INTERESTED IN THIS ISSUE, PLEASE SEE [THIS COMMENT](https://github.com/ericmjl/pyjanitor/issues/612#issuecomment-557250963) FOR PROPOSED CODE.

------

# Brief Description

`clean_names` method does not work when an integer is used as column name 

# Minimally Reproducible Code


```
rankings = {
    "countries_to_play_cricket": [
        "India",
        "South Africa",
        "England",
        "New Zealand",
        "Australia",
    ],
    0: ["England", "India", "New Zealand", "South Africa", "Pakistan"],
    "t20": ["Pakistan", "India", "Australia", "England", "New Zealand"],
}

rankings_pd = pd.DataFrame(rankings)

df = rankings_pd.clean_names()
```

# Error Messages

```
AttributeError: 'int' object has no attribute 'lower'
```

If I change the column name integer 0 to string "0" then there will be no error. But in that case it defeats the purpose of using a method :)


Discussion:
This is similar to #564 , which seems to have gone a bit stale at this point.
@shantanuo I'm copying in your email reply here so that it's on the thread for others to review (in italics):

_Yes. This is exact duplicate of the bug #564 
But I did not understand what "stale" mean. Is it that difficult to convert the integers to strings in python?<br>
"I feel like because non-string column names are supported by pandas, pyjanitor should support that capability."<br>
I do not agree with this approach. There are a lot of things pandas is not doing correctly. For e.g. issue #70 deals with invalid parameters in methods like read_excel. Pandas simply ignores the extra parameters without throwing an exception. It leads to lot of confusion. We should fix this, that and more!_

In terms of "stale," I was simply indicating that issue #564 hasn't had a comment response in over a month.  So, it's not clear right now who (if anyone) is taking charge on it (though it does appear from his comment that @dendrondal may have started on some stuff).

@ericmjl I suggest we drop this issue and merge it in with #564.  We can continue the conversation there.  But, I'm not sure the best way to do that in GitHub (and I'm guessing that yourself or another collaborator would be the only ones with the power to do it).
#610 should have fixed this issue. I'll see later this evening if I can find out what's causing this issue. Either way, this should definitely be merged with #564 . 

@shantanuo , I agree with your sentiment on Pandas' approach to typing. I think it should enforce strictly string-based column names, automatically doing the conversion when the dataframe is instantiated. However, I believe the sentiment with PyJanitor is that it is meant to be a direct drop-in replacement for Pandas, meaning the same typing should be used, allowing for easy transfer of code between the two.

I briefly mentioned automatic conversion of column names to strings in #564 . I believe the quickest way to do this would be to create a wrapper around pd.DataFrame which does this, which would also allow customization of dataframe attributes down the road, if needed (i.e. for compatibility with PySpark, etc). @ericmjl , I'd love to hear your thoughts on this, as well as any other features you think would be a good inclusion to a janitor.DataFrame class.
@dendrondal and @rahosbach, thank you for chiming in in my absence. Apologies, work has taken over many hours of my day.

First off, to address the main issue: yes, we recently merged #610, though the implementation has not changed, hence the error message still shows up. To solve this, I am comfortable with accepting a PR that enforces string conversion inside of `clean_names` for every column name before name cleaning, as long as the behaviour is documented in the docstring and justifies the design choice. After all, some people will have integer column names, and to have them suddenly (and more importantly, _silently_) changed to strings is going to surprise people. The fewer the surprises the better. @shantanuo, this would be a one-line code change + docstring change + test addition, and I would love for this to be another open source contribution from you.

Now, onto some bigger issues.

The biggest one is the relationship between `pyjanitor` and `pandas`. I started it initially as a port of the `janitor` package, but as people used it and the user base grew, maybe some confusions w.r.t. the API policy. I've had it in my head for a long time, but never wrote it down formally, so this is something that should be on me to do, but I'll state the summary below:

- pyjanitor only extends or aliases the pandas API (and other dataframe APIs), but will never fix or replace them.
- Undesirable pandas behaviour should be reported upstream in the `pandas` repository.
- If at some point the `pandas` devs decide to take something from `pyjanitor` and internalize it as part of the official `pandas` API, then we will deprecate it from `pyjanitor`, while acknowledge the original contributors' contribution as part of the official deprecation record.

(If others have ideas and/or would like to officially make an API Policy section/file in our docs by first copy/pasting the above bullet points, happy to review the PR!)

Secondly, on why there's no `janitor.DataFrame` class. Actually, it did exist before, but I scrapped it once pandas-flavor came along. Subclassing the DataFrame is quite tricky to maintain, and resulted in code duplication, while pandas-flavor allowed us to avoid duplication of function names. The philosophical/design choice here was that the fewer objects with state to maintain, the better. As I've learned, functions > objects for most applications that I've encountered.

Also, big thanks to @rahosbach for addressing @shantanuo's email. I do prefer that the discussion stay on the issue tracker, so that no contributor feels overburdened to do software support alone, so what you did, @rahosbach, was the right thing in my view! Centralizing the communication channels here where the public record is visible also helps others learn about the development of the project.

Now, for some housekeeping business.

#564 - I think @dendrondal has done a tremendous job with the PR #610. So I'm going to close that one. Yes, the `clean_names` bug is still present :smile:, but hear me out.

I'm going to change the name of this issue, to propose that `clean_names` does automagic string conversion. The line change is simple, so anyone who wants to take this can do so (I haven't had the bandwidth to do a full PR, but am happy to give away the contribution to others):

```python
# if we want mapping back to originals:
new_cols = []
mapping = {}
for c in df.columns:
    new_cols.append(str(c))
    mapping[str(c)] = c
# Then use the mapping to convert back to original column name, maybe?
```

And then we'd probably want some docstrings explaining why there's conversion to strings + no conversion back to original types... unless someone wants to take that too! :smile:

I'm going to end my posting here, and get back to work :).

Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -18,6 +18,7 @@
     Union,
 )
 
+
 import numpy as np
 import pandas as pd
 import pandas_flavor as pf
@@ -226,12 +227,16 @@ def clean_names(
     remove_special: bool = False,
     strip_accents: bool = True,
     preserve_original_columns: bool = True,
+    enforce_string: bool = True,
 ) -> pd.DataFrame:
     """
     Clean column names.
 
-    Takes all column names, converts them to lowercase, then replaces all
-    spaces with underscores.
+    Takes all column names, converts them to lowercase,
+    then replaces all spaces with underscores.
+
+    By default, column names are converted to string types.
+    This can be switched off by passing in ``enforce_string=False``.
 
     This method does not mutate the original DataFrame.
 
@@ -270,10 +275,16 @@ def clean_names(
         Only letters, numbers and underscores are preserved.
     :param preserve_original_columns: (optional) Preserve original names.
         This is later retrievable using `df.original_columns`.
+    :param enforce_string: Whether or not to convert all column names
+        to string type. Defaults to True, but can be turned off.
+        Columns with >1 levels will not be converted by default.
     :returns: A pandas DataFrame.
     """
     original_column_names = list(df.columns)
 
+    if enforce_string:
+        df = df.rename(columns=lambda x: str(x))
+
     df = df.rename(columns=lambda x: _change_case(x, case_type))
 
     df = df.rename(columns=_normalize_1)


Test patch:
diff --git a/tests/functions/test_clean_names.py b/tests/functions/test_clean_names.py
--- a/tests/functions/test_clean_names.py
+++ b/tests/functions/test_clean_names.py
@@ -161,3 +161,9 @@ def test_clean_names_camelcase_to_snake(dataframe):
         "snakes_on_a_plane2",
         "snakes_on_a_plane3",
     ]
+
+
+def test_clean_names_enforce_string(dataframe):
+    df = dataframe.rename(columns={"a": 1}).clean_names(enforce_string=True)
+    for c in df.columns:
+        assert isinstance(c, str)