Repository: pyjanitor-devs/pyjanitor

Issue:
[TST] Test coverage of functions (and lines) needs to improve
While updating function argument names, I came across a function without a test, and was curious what other functions didn't have tests. I ran the `pytest coverage` plugin and got the following results. 

```
----------- coverage: platform win32, python 3.6.7-final-0 -----------
Name                                  Stmts   Miss  Cover   Missing
-------------------------------------------------------------------
janitor\__init__.py                       2      0   100%
janitor\biology.py                       13      2    85%   12-13
janitor\chemistry.py                     69     48    30%   15, 64, 106-124, 164-186, 229-275, 307-318
janitor\errors.py                         2      0   100%
janitor\finance.py                       41     12    71%   53, 76-89, 101, 196
janitor\functions.py                    520     66    87%   606-614, 1113, 1139, 1143, 1192, 1198, 1468, 1817, 2061-2066, 2218-2224, 2235-2263, 2267-2274, 2278-2279, 2463-2499, 2590, 2609, 2895
janitor\io.py                            16      0   100%
janitor\testing_utils\__init__.py         0      0   100%
janitor\testing_utils\date_data.py        1      0   100%
janitor\testing_utils\strategies.py      12      1    92%   11
janitor\utils.py                         24      0   100%
-------------------------------------------------------------------
TOTAL                                   700    129    82%
```

This tells us which lines are not covered, but more importantly I wanted to know which functions have no coverage at all. Does anyone know how to get that answer? For the most part, the large blocks of code are functions that have no coverage:

```
chemistry
    smiles2mol
    morgan_fingerprint
    molecular_descriptors
    maccs_keys_fingerprint

functions
    _clean_accounting_column
    _currency_column_to_numeric
    _replace_empty_string_with_none
    currency_column_to_numeric
```

Overall, we have a test for at least some part of most functions, but `chemistry` doesn't have any, and there are a few in the main `functions` that are lacking.

This issue serves to identify functions that folks can write tests for, and discuss coverage.
[DOC] bin_numeric is missing from the docs "General Functions" page
As per title.
[DOC] Produce notebook showing how to normalize and standardize columns using pyjanitor
I like to add a normalize and standardize function to pyjanitor. I guess we can do this with transform function, but I like the readability of the code over a lambda function to make it work. Like normalize_columns() or standardize_columns(), since we already have expand_columns() (aka. dummy codes) and max_min_scale(). This will put preprocessing into the hands of more beginners, even though I have nothing against sklearn. I will say, doing a .fit() call on preprocessing bothers me, and I dont know why? What are your thoughts?


Discussion:


@jcvall I debated a bit with myself when thinking about whether to put min_max_scale and expand_columns inside. I think it's cool to put in `normalize`/`standardize`, but I think that it (and now that I'm thinking about it, `min_max_scale` as well) should come with a warning in the docs, that state that any transformations that involve a summary statistic of a dataset (e.g. mean, median, max, min, etc.) will result in testing set "contamination", or "leakage of information from training set into testing set". For this reason, sklearn's preprocessing is instead more principled.

There is one setting where I think having those functions could be useful though - in Bayesian modelling, when we are already sure of the model structure, then fitting the model necessarily means using the entire dataset, hence no train/test splitting happens. Here, having these functions could be useful as well.

Even in light of these statistical considerations, I would encourage you to submit the PR! We can probably do it with the right caveats written into the docstring, so that other end-users are aware of how to not abuse the convenience functionality. What do you think?
I actually didn't think about leakage in machine learning.  I guess what you are saying is that when you perform the preprocessing  you can split your data; however, the deviations from the mean for normalized data will be in both the training and test set.  So the mean is inferred in both data sets.  If I am thinking correctly, then that is a problem...Let me know if I am off base here.  I guess in Bayesian and more classical statistics (linear regression) it is rather helpful function for the coefficients, so I see your point.  Okay.  I will go ahead  knowing that this should be used with caution.  
Yep, you're on the right track there!
Sorry, you don't have to answer this right away, but would the sklearn-pandas package be helpful here?  

Example from GitHub page:
`Outputting a dataframe

By default the output of the dataframe mapper is a numpy array. This is so because most sklearn estimators expect a numpy array as input. If however we want the output of the mapper to be a dataframe, we can do so using the parameter df_out when creating the mapper:

```
mapper_df = DataFrameMapper([
('pet', sklearn.preprocessing.LabelBinarizer()),
['children'], sklearn.preprocessing.StandardScaler())
], df_out=True)
np.round(mapper_df.fit_transform(data.copy()), 2)
   pet_cat  pet_dog  pet_fish  children
0        1        0         0      0.21
1        0        1         0      1.88
2        0        1         0     -0.63
3        0        0         1     -0.63
4        1        0         0     -1.46
5        0        1         0     -0.63
6        1        0         0      1.04
7        0        0         1      0.21`
```
I gave it some thought last night and this morning.

It could be good! We already use scikit-learn underneath the hood, I think. 

One thing I did was to profile how fast using sklearn was compared to a custom implementation of one-hot encoding, and I found my custom implementation was faster. That said, for convenience, I ended up just using the get_dummies function provided by pandas. Maybe measuring function speed could be helpful here in deciding which implementation to use?
I am sort of now leaning toward doing the documentation of both methods.  One with the normalize_columns and standardize_columns with Bayesian or classical statistics and another doing machine learning.  However, I am going to study how the mlxtend package handles this and see what I can learn. 

Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -1183,6 +1183,9 @@ def filter_column_isin(
 
         df = df[df['names'].isin(['James', 'John'])]
 
+    If "complement" is true, then we will only get rows for which the names
+    are not James or John.
+
     :param df: A pandas DataFrame
     :param column_name: The column on which to filter.
     :param iterable: An iterable. Could be a list, tuple, another pandas
@@ -2241,7 +2244,11 @@ def currency_column_to_numeric(
     :param df: The DataFrame
     :param column_name: The column to modify
     :param cleaning_style: What style of cleaning to perform. If None, standard
-        cleaning is applied. Options are: 'accounting'.
+        cleaning is applied. Options are:
+
+            * 'accounting':
+            Replaces numbers in parentheses with negatives, removes commas.
+
     :param cast_non_numeric: A dict of how to coerce certain strings. For
         example, if there are values of 'REORDER' in the DataFrame,
         {'REORDER': 0} will cast all instances of 'REORDER' to 0.


Test patch:
diff --git a/tests/chemistry/test_morgan_fingerprint.py b/tests/chemistry/test_morgan_fingerprint.py
--- a/tests/chemistry/test_morgan_fingerprint.py
+++ b/tests/chemistry/test_morgan_fingerprint.py
@@ -29,3 +29,15 @@ def test_morgan_fingerprint_bits(chemdf):
     )
     assert morgans.shape == (10, 2048)
     assert set(morgans.values.flatten().tolist()) == set([0, 1])
+
+
+@pytest.mark.skipif(
+    importlib.util.find_spec("rdkit") is None,
+    reason="rdkit tests only required for CI",
+)
+@pytest.mark.chemistry
+def test_morgan_fingerprint_kind_error(chemdf):
+    with pytest.raises(ValueError):
+        chemdf.smiles2mol("smiles", "mol").morgan_fingerprint(
+            "mol", kind="hdfs"
+        )
diff --git a/tests/finance/test_convert_currency.py b/tests/finance/test_convert_currency.py
--- a/tests/finance/test_convert_currency.py
+++ b/tests/finance/test_convert_currency.py
@@ -1,7 +1,8 @@
 import pytest
 import requests
+from datetime import date, datetime
 
-import janitor.finance
+from janitor.finance import convert_currency
 
 
 @pytest.mark.finance
@@ -26,3 +27,33 @@ def test_make_currency_api_request():
 def test_make_new_currency_col(dataframe):
     df = dataframe.convert_currency("a", "USD", "USD", make_new_column=True)
     assert all(df["a"] == df["a_USD"])
+
+
+@pytest.mark.finance
+def test_historical_datetime(dataframe):
+    with pytest.raises(ValueError):
+        assert dataframe.convert_currency(
+            "a",
+            "USD",
+            "AUD",
+            make_new_column=True,
+            historical_date=datetime(1982, 10, 27),
+        )
+
+
+@pytest.mark.finance
+def test_historical_date(dataframe):
+    with pytest.raises(ValueError):
+        assert dataframe.convert_currency(
+            "a",
+            "USD",
+            "AUD",
+            make_new_column=True,
+            historical_date=date(1982, 10, 27),
+        )
+
+
+@pytest.mark.finance
+def test_currency_check(dataframe):
+    with pytest.raises(ValueError):
+        assert dataframe.convert_currency("a", "USD", "ASDF")
diff --git a/tests/utils/test_clean_accounting_column.py b/tests/utils/test_clean_accounting_column.py
new file mode 100644
--- /dev/null
+++ b/tests/utils/test_clean_accounting_column.py
@@ -0,0 +1,14 @@
+import pytest
+from janitor.utils import _clean_accounting_column
+
+
+@pytest.mark.functions
+def test_clean_accounting_column():
+    test_str = "(1,000)"
+    assert _clean_accounting_column(test_str) == float(-1000)
+
+
+@pytest.mark.functions
+def test_clean_accounting_column_zeroes():
+    test_str = "()"
+    assert _clean_accounting_column(test_str) == 0.00
diff --git a/tests/utils/test_currency_column_to_numeric.py b/tests/utils/test_currency_column_to_numeric.py
new file mode 100644
--- /dev/null
+++ b/tests/utils/test_currency_column_to_numeric.py
@@ -0,0 +1,22 @@
+import pytest
+from janitor.utils import _currency_column_to_numeric
+
+
+@pytest.mark.function
+def test_empty_input():
+    assert _currency_column_to_numeric("") == "ORIGINAL_NA"
+
+
+@pytest.mark.function
+def test_cast_non_numeric_true():
+    assert _currency_column_to_numeric("foo", {"foo": 42}) == 42
+
+
+@pytest.mark.function
+def test_cast_non_numeric_false():
+    assert _currency_column_to_numeric("10 dollars", {"foo": 42}) == "10"
+
+
+@pytest.mark.function
+def test_non_cast_input():
+    assert _currency_column_to_numeric("-1,000,000 yen") == "-1000000"
diff --git a/tests/functions/test_idempotent.py b/tests/utils/test_idempotent.py
similarity index 100%
rename from tests/functions/test_idempotent.py
rename to tests/utils/test_idempotent.py
diff --git a/tests/utils/test_import_message.py b/tests/utils/test_import_message.py
new file mode 100644
--- /dev/null
+++ b/tests/utils/test_import_message.py
@@ -0,0 +1,15 @@
+import pytest
+from janitor.utils import import_message
+
+
+@pytest.mark.function
+def test_import_message(capsys):
+    message = (
+        "To use the janitor submodule chemistry, you need to install "
+        "RDKit.\n\n"
+        "To do so, use the following command:\n\n"
+        "    conda install -c rdkit rdkit\n"
+    )
+    import_message("chemistry", "RDKit", "conda install -c rdkit rdkit")
+    captured = capsys.readouterr()
+    assert captured.out == message
diff --git a/tests/utils/test_replace_empty_string_with_none.py b/tests/utils/test_replace_empty_string_with_none.py
new file mode 100644
--- /dev/null
+++ b/tests/utils/test_replace_empty_string_with_none.py
@@ -0,0 +1,26 @@
+import pytest
+from janitor.utils import (
+    _replace_empty_string_with_none,
+    _replace_original_empty_string_with_none,
+)
+
+
+@pytest.mark.function
+def test_int_input():
+    assert _replace_empty_string_with_none(4) == 4
+
+
+@pytest.mark.function
+def test_float_input():
+    assert _replace_empty_string_with_none(4.0) == 4.0
+
+
+@pytest.mark.function
+def test_empty_string_input():
+    assert _replace_empty_string_with_none("") is None
+
+
+@pytest.mark.function
+def test_replace_original_empty_string_with_none():
+    assert _replace_original_empty_string_with_none("foo") == "foo"
+    assert _replace_original_empty_string_with_none("ORIGINAL_NA") is None