Repository: pyjanitor-devs/pyjanitor

Issue:
data_dictionary() function
I wanted an easy way to create a data dictionary.....I came up with this...

![image](https://user-images.githubusercontent.com/20308031/57188711-757f2d80-6ed1-11e9-94dc-2e3dc93c1340.png)

![image](https://user-images.githubusercontent.com/20308031/57188717-82038600-6ed1-11e9-9a35-8e209a9651da.png)



Discussion:
I think this could be cool! It wouldn't be a data cleaning function, but could be one of the "utilities". I like where this function is going, and it looks like it could be expandable as well.
If anyone at sprints wants to take this one on, I have no objections!  If they don't
 no worries.  I will put a PR in next week.  I know they will love pyjanitor as much as I do.
Great idea!

Just some initial thoughts:

1. If you create a new `description` object for each column and you overload `__doc__` within said object, then you can run `help(df.column.description)` to get the column description. I'm not sure exactly how it would be implemented, but that's the general idea. The following works, but seems unstable if you want to use the same `class` over and over again.

```python
class description:
	__doc__ = 'Some description'

column = description()
help(column)
 ```

2. Might be useful to have all this information formatted as some sort of webpage. I.e. `df.data_dictionary()` would open up a browser with all the information outside of `jupyter` or whatever instance the user is running `pyjanitor`.
@szuckerman this is a super creative idea! At first glance, I especially like the idea of overwriting the `__doc__` attribute for a new dataframe. 

Perhaps one appropriate way of using this `data_dictionary` function would be instead to load a YAML file that is the data dictionary? Desired API/example below:

```python
# Create dataframe and describe data with a YAML file
df = pd.DataFrame(...).describe_data('data_dictionary.yml')  # with more methods chained afterwards

# Alternatively, create dataframe and describe data with a dictionary.
data_dict = {
    "col1": "description1",
    "col2": "description2",
    ...
}
df = pd.DataFrame(...).describe_data(data_dict)
```

Thinking more about it... Underneath the hood, what `describe_data` could do is to register a custom accessor, called `data_dictionary`, with methods such as `print()`/`display()` (to send a representation to the screen), `update()` (to update the data dictionary), `validate()` (to report on columns missing a description, etc.), and perhaps more. 

Custom accessors might be preferred over overriding `__doc__`, as this is favoured by `pandas`. Functionality to do this is provided in `pandas_flavor` as well: https://github.com/Zsailer/pandas_flavor (scroll to the bottom of the README).

I'm going to tag this one as a good intermediate issue.
I will take a crack at this ticket


Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -2725,6 +2725,57 @@ def groupby_agg(
     return df
 
 
+@pf.register_dataframe_accessor("data_description")
+class DataDescription:
+    """
+    Accessor that provides high-level description of data present
+    in this DataFrame.
+    """
+
+    def __init__(self, data):
+        self._data = data
+        self._desc = dict()
+
+    def _get_data_df(self):
+        df = self._data
+
+        data_dict = dict()
+        data_dict["column_name"] = df.columns.tolist()
+        data_dict["type"] = df.dtypes.tolist()
+        data_dict["count"] = df.count().tolist()
+        data_dict["pct_missing"] = (1 - (df.count() / len(df))).tolist()
+        data_dict["description"] = [self._desc.get(c, "") for c in df.columns]
+
+        return pd.DataFrame(data_dict).set_index("column_name")
+
+    @property
+    def df(self):
+        """
+        Get a table of descriptive information in a DataFrame format.
+        """
+        return self._get_data_df()
+
+    def display(self):
+        """
+        Print the table of descriptive information about this DataFrame.
+        """
+        print(self._get_data_df())
+
+    def set_description(self, desc: Union[List, Dict]):
+        """
+        Update the description for each of the columns in the DataFrame.
+
+        :param desc: The structure containing the descriptions to update
+        :type desc: list or dict
+        """
+        if isinstance(desc, list):
+            assert len(desc) == len(self._data.columns)
+            self._desc = dict(zip(self._data.columns, desc))
+
+        elif isinstance(desc, dict):
+            self._desc = desc
+
+
 @pf.register_dataframe_method
 def drop_duplicate_columns(
     df: pd.DataFrame, column_name: str, nth_index: int = 0


Test patch:
diff --git a/tests/functions/test_data_description.py b/tests/functions/test_data_description.py
new file mode 100644
--- /dev/null
+++ b/tests/functions/test_data_description.py
@@ -0,0 +1,100 @@
+import numpy as np
+import pandas as pd
+import pytest
+
+
+@pytest.mark.functions
+def test_no_descriptions(dataframe):
+    df = dataframe.data_description.df
+    assert df.index.name == "column_name"
+    assert (df["description"].str.len() == 0).all()
+    assert (df["count"] == len(dataframe)).all()
+    assert (df["pct_missing"] == 0).all()
+
+
+@pytest.mark.functions
+def test_description_list(dataframe):
+    desc = ["A", "B", "C", "D", "E"]
+
+    dataframe.data_description.set_description(desc)
+    df = dataframe.data_description.df
+
+    assert (df["description"] == desc).all()
+
+    with pytest.raises(AssertionError):
+        dataframe.data_description.set_description([])
+
+    with pytest.raises(AssertionError):
+        dataframe.data_description.set_description(desc[0:3])
+
+
+@pytest.mark.functions
+def test_description_full_dict(dataframe):
+    desc = {
+        "a": "First",
+        "Bell__Chart": "Second",
+        "decorated-elephant": "Third",
+        "animals@#$%^": "Fourth",
+        "cities": "Fifth",
+    }
+
+    dataframe.data_description.set_description(desc)
+    df = dataframe.data_description.df
+
+    assert not (df["description"].str.len() == 0).any()
+
+    for k, v in desc.items():
+        assert df.loc[k]["description"] == v
+
+
+@pytest.mark.functions
+def test_description_partial_dict(dataframe):
+    desc = {"a": "First", "decorated-elephant": "Third", "cities": "Fifth"}
+
+    dataframe.data_description.set_description(desc)
+    df = dataframe.data_description.df
+
+    assert len(df[df["description"].apply(lambda x: len(x) == 0)]) == 2
+
+    for k, v in desc.items():
+        assert df.loc[k]["description"] == v
+
+
+@pytest.mark.functions
+def test_null_rows(dataframe):
+    dataframe = dataframe.copy()
+    dataframe.loc[1] = None
+    dataframe.loc[4] = None
+    dataframe.loc[6] = None
+
+    df = dataframe.data_description.df
+    assert (df["count"] == 6).all()
+    assert np.isclose(df["pct_missing"], (3 / 9)).all()
+
+
+@pytest.mark.functions
+def test_null_values(dataframe):
+    dataframe = dataframe.copy()
+    dataframe.loc[1, "a"] = None
+    dataframe.loc[4, "a"] = None
+    dataframe.loc[3, "cities"] = None
+    dataframe.loc[3, "Bell__Chart"] = None
+    dataframe.loc[6, "decorated-elephant"] = None
+
+    df = dataframe.data_description.df
+    assert df.loc["a", "count"] == 7
+    assert np.isclose(df.loc["a", "pct_missing"], 2 / 9)
+
+    assert df.loc["cities", "count"] == 8
+    assert np.isclose(df.loc["cities", "pct_missing"], 1 / 9)
+
+    assert df.loc["Bell__Chart", "count"] == 8
+    assert np.isclose(df.loc["Bell__Chart", "pct_missing"], 1 / 9)
+
+    assert df.loc["decorated-elephant", "count"] == 8
+    assert np.isclose(df.loc["decorated-elephant", "pct_missing"], 1 / 9)
+
+
+@pytest.mark.functions
+def test_display(dataframe):
+    dataframe.data_description.display()