Repository: pyjanitor-devs/pyjanitor
Pull request #977 (2022-01-07 02:13:06)

[ENH] Add I/O function read_commandline()

# PR Description

First order of business: No automated tests, sadly. Not because I didn't feel like it or didn't want to- but rather pytest was giving me trouble and I wanted to get a bit of feedback before wasting too much time. It took me longer than I'd like to admit to throw in the towel on that one. 

Testing WOULD be done with the included  ```manual_read_command(string:str)``` in case any body wants to take up the mantle with that real quick.

Second: I did manual testing using a publicly available .csv spreadsheet of data, which I have named sep-quarter.csv. For those unfamiliar: ```argv``` is used to represent the commandline arguments. ```argv[0]``` is always the filename (thus ```python``` or ```python3``` are not included in the argument list) and everything from ```argv[1]``` until end-of-line is an additional command.

I have tested the following use cases:

``` python3 <filename>.py cat sep-quarter.csv ``` creates a pandas df from the .csv file, which in my testing is a big dataframe. About 19000 rows.

 ``` python3 <filename>.py 'cat sep-quarter.csv | grep '.SEA1AA' ' ```  creates a pandas df only from the infromation in the .csv file that contains .SEA1AA, which in my testing results in 41 rows. Much smaller, and only containing information relevant to what was queried, in this case, anything with .SEA1AA in one of its columns.

 NOTE: the entirety of argv[1] and beyond needs to be in a string for this, it doesn't work when it's not in quotes. The command I mentioned above this one however does work, with or without quotes. Unfortunately, when working with the shell there's always the possibility of a conflict: I think the ``` | grep ... ``` line tells the shell that the grep is a different command, so the shell will execute ```python 3 <filename>.py cat sep-quarter.csv ``` and then ``` grep '.SEA1AA' ``` separately, which naturally yields an incorrect result for the df.

```python3 <filename.py>  'unzip -cq sep-quarter.csv.zip' ``` (in quotes for the same reason as above): Will unzip the .csv file before creating a df out of the contents within. This creates an identical df to the first command I mentioned.

I have included the .csv files in the branch, both zipped and unzipped. Those can be moved deleted at will, they're just there for testing.

**This PR lays the foundation for resolving #945 **

# Relevant Reviewers

<!-- Finally, please tag relevant maintainers to review. -->

Please tag maintainers to review.

- @ericmjl


codecov[bot]: # [Codecov](https://codecov.io/gh/pyjanitor-devs/pyjanitor/pull/977?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pyjanitor-devs) Report
> Merging [#977](https://codecov.io/gh/pyjanitor-devs/pyjanitor/pull/977?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pyjanitor-devs) (3fa49f2) into [dev](https://codecov.io/gh/pyjanitor-devs/pyjanitor/commit/d6ef2c341d24bb083fc4538d0717b7f40386a095?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pyjanitor-devs) (d6ef2c3) will **decrease** coverage by `0.46%`.
> The diff coverage is `60.00%`.


```diff
@@            Coverage Diff             @@
##              dev     #977      +/-   ##
==========================================
- Coverage   96.30%   95.84%   -0.47%     
==========================================
  Files          77       78       +1     
  Lines        2817     2818       +1     
==========================================
- Hits         2713     2701      -12     
- Misses        104      117      +13     
```

BaritoneBeard: Heard, thank you both.

I'll accept some or all changes suggested by @ericmjl and get to work refactoring to look more in line with what @samukweku has laid out. A word of 'warning' is that I tend to work slowly on stuff I'm not being paid for (because I'm tired from working on stuff I do get paid for), I'll probably take a few days for this.
ericmjl: Not a problem! We’re all volunteers here on this project, so it’s
totally reasonable :). 

Can I check, did you enable allowing maintainers to push to the PR
branch? If so, I might help move some things along, though I’m also busy
with another PR to the repo. 

BaritoneBeard: According to the little check box, moderators are allowed access and edit, so they should be allowed to push, correct? 
ericmjl: I think that's right! Thank you for confirming, @BaritoneBeard!

Patch:
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
index 1875a0d..2898e00 100644
--- a/.devcontainer/devcontainer.json
+++ b/.devcontainer/devcontainer.json
@@ -10,11 +10,23 @@
 	},
 	// Set *default* container specific settings.json values on container create.
 	"settings": {
-		"terminal.integrated.shell.linux": "/bin/bash",
+		"terminal.integrated.defaultProfile.linux": "bash",
 		"python.pythonPath": "/opt/conda/bin/python",
 		"python.linting.enabled": true,
 		"python.linting.pylintEnabled": true,
-		"python.linting.pylintPath": "/opt/conda/bin/pylint"
+		"python.linting.pylintPath": "/opt/conda/bin/pylint",
+		"python.formatting.provider": "black",
+		"python.formatting.blackArgs": [
+			"--config",
+			"pyproject.toml",
+		],
+		"editor.formatOnSave": true,
+		"files.insertFinalNewline": true,
+		"files.trimFinalNewlines": true,
+		"files.trimTrailingWhitespace": true,
+		"[python]": {
+			"editor.formatOnSaveMode": "file",
+		},
 	},
 	// Add the IDs of extensions you want installed when the container is created.
 	"extensions": [
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d164b73..31e93be 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,7 @@
 -   [ENH] Deprecate `num_bins` from `bin_numeric` in favour of `bins`, and allow generic `**kwargs` to be passed into `pd.cut`. Issue #969. @thatlittleboy
 -   [ENH] Fix `concatenate_columns` not working on category inputs @zbarry
 -   [INF] Simplify CI system @ericmjl
+-   [ENH] Added "read_commandline" function to janitor.io @BaritoneBeard
 
 ## [v0.22.0] - 2021-11-21
 
diff --git a/environment-dev.yml b/environment-dev.yml
index 259849c..fffa731 100644
--- a/environment-dev.yml
+++ b/environment-dev.yml
@@ -2,7 +2,7 @@ name: pyjanitor-dev
 channels:
   - conda-forge
 dependencies:
-  - python=3.7
+  - python=3.9
   - biopython
   - black=21.7b0
   - bump2version=1.0.1
diff --git a/janitor/functions/case_when.py b/janitor/functions/case_when.py
index 8b7f429..1012229 100644
--- a/janitor/functions/case_when.py
+++ b/janitor/functions/case_when.py
@@ -10,10 +10,40 @@ from janitor.utils import check
 @pf.register_dataframe_method
 def case_when(df: pd.DataFrame, *args, column_name: str) -> pd.DataFrame:
     """
-    Convenience function for creating a column,
-    based on a condition, or multiple conditions.
-
-    It is similar to SQL and dplyr's case_when,
+    Create a column based on a condition or multiple conditions.
+
+    Example usage:
+
+        >>> import pandas as pd
+        >>> import janitor
+        >>> df = pd.DataFrame(
+        ...     {
+        ...         "a": [0, 0, 1, 2, "hi"],
+        ...         "b": [0, 3, 4, 5, "bye"],
+        ...         "c": [6, 7, 8, 9, "wait"],
+        ...     }
+        ... )
+        >>> df
+            a    b     c
+        0   0    0     6
+        1   0    3     7
+        2   1    4     8
+        3   2    5     9
+        4  hi  bye  wait
+        >>> df.case_when(
+        ...     ((df.a == 0) & (df.b != 0)) | (df.c == "wait"), df.a,
+        ...     (df.b == 0) & (df.a == 0), "x",
+        ...     df.c,
+        ...     column_name="value",
+        ... )
+            a    b     c value
+        0   0    0     6     x
+        1   0    3     7     0
+        2   1    4     8     8
+        3   2    5     9     9
+        4  hi  bye  wait    hi
+
+    Similar to SQL and dplyr's case_when
     with inspiration from `pydatatable` if_else function.
 
     If your scenario requires direct replacement of values,
@@ -57,38 +87,7 @@ def case_when(df: pd.DataFrame, *args, column_name: str) -> pd.DataFrame:
         default
     ```
 
-    Functional usage syntax:
-
-    ```python
-    import pandas as pd
-    import janitor as jn
-
-    df = pd.DataFrame(...)
-    right = pd.DataFrame(...)
-
-    df = jn.case_when(
-        df,
-        condition0, result0,
-        condition1, result1,
-        ...,
-        default,
-        column_name = 'column',
-    )
-    ```
-
-    Method chaining syntax:
-
-    ```python
-    df = df.case_when(
-        condition0, result0,
-        condition1, result1,
-        ...,
-        default,
-        column_name = 'column',
-    )
-    ```
-
-    :param df: A Pandas dataframe.
+    :param df: A pandas DataFrame.
     :param args: Variable argument of conditions and expected values.
         Takes the form
         `condition0`, `value0`, `condition1`, `value1`, ..., `default`.
@@ -102,7 +101,7 @@ def case_when(df: pd.DataFrame, *args, column_name: str) -> pd.DataFrame:
         For a 1-D array, it should have the same length as the DataFrame.
         The `default` argument applies if none of `condition0`,
         `condition1`, ..., evaluates to `True`.
-        Value can be a scalar, a callabe, or a 1-D array. if `default` is a
+        Value can be a scalar, a callable, or a 1-D array. if `default` is a
         callable, it should evaluate to a 1-D array.
         The 1-D array should be the same length as the DataFrame.
     :param column_name: Name of column to assign results to. A new column
diff --git a/janitor/functions/clean_names.py b/janitor/functions/clean_names.py
index c45a89e..37c0a1c 100644
--- a/janitor/functions/clean_names.py
+++ b/janitor/functions/clean_names.py
@@ -30,23 +30,32 @@ def clean_names(
 
     This method does not mutate the original DataFrame.
 
-    Functional usage syntax:
-
-    ```python
-    df = clean_names(df)
-    ```
-
-    Method chaining syntax:
-
-    ```python
-    import pandas as pd
-    import janitor
-    df = pd.DataFrame(...).clean_names()
-
-    Examples of transformation:
+    Example usage:
 
-    - Columns before: First Name, Last Name, Employee Status, Subject
-    - Columns after: first_name, last_name, employee_status, subject
+        >>> import pandas as pd
+        >>> import janitor
+        >>> df = pd.DataFrame(
+        ...     {
+        ...         "Aloha": range(3),
+        ...         "Bell Chart": range(3),
+        ...         "Animals@#$%^": range(3)
+        ...     }
+        ... )
+        >>> df
+           Aloha  Bell Chart  Animals@#$%^
+        0      0           0             0
+        1      1           1             1
+        2      2           2             2
+        >>> df.clean_names()
+           aloha  bell_chart  animals@#$%^
+        0      0           0             0
+        1      1           1             1
+        2      2           2             2
+        >>> df.clean_names(remove_special=True)
+           aloha  bell_chart  animals
+        0      0           0        0
+        1      1           1        1
+        2      2           2        2
 
     :param df: The pandas DataFrame object.
     :param strip_underscores: (optional) Removes the outer underscores from all
diff --git a/janitor/functions/sort_naturally.py b/janitor/functions/sort_naturally.py
index a07da6d..35a0e80 100644
--- a/janitor/functions/sort_naturally.py
+++ b/janitor/functions/sort_naturally.py
@@ -1,3 +1,4 @@
+"""Implementation of the `sort_naturally` function."""
 import pandas_flavor as pf
 import pandas as pd
 from natsort import index_natsorted
@@ -39,32 +40,33 @@ def sort_naturally(
     after the column name to sort by is provided.
     They are passed through to the `natsorted` function.
 
-    Functional usage syntax:
+    Example:
+
+        >>> import pandas as pd
+        >>> import janitor
+        >>> df = pd.DataFrame(
+        ...     {
+        ...         "Well": ["A21", "A3", "A21", "B2", "B51", "B12"],
+        ...         "Value": [1, 2, 13, 3, 4, 7],
+        ...     }
+        ... )
+        >>> df
+          Well  Value
+        0  A21      1
+        1   A3      2
+        2  A21     13
+        3   B2      3
+        4  B51      4
+        5  B12      7
+        >>> df.sort_naturally("Well")
+          Well  Value
+        1   A3      2
+        0  A21      1
+        2  A21     13
+        3   B2      3
+        5  B12      7
+        4  B51      4
 
-    ```python
-        import pandas as pd
-        import janitor as jn
-
-        df = pd.DataFrame(...)
-
-        df = jn.sort_naturally(
-            df=df,
-            column_name='alphanumeric_column',
-        )
-    ```
-
-    Method chaining usage syntax:
-
-    ```python
-        import pandas as pd
-        import janitor
-
-        df = pd.DataFrame(...)
-
-        df = df.sort_naturally(
-            column_name='alphanumeric_column',
-        )
-    ```
     :param df: A pandas DataFrame.
     :param column_name: The column on which natural sorting should take place.
     :param natsorted_kwargs: Keyword arguments to be passed
diff --git a/janitor/functions/take_first.py b/janitor/functions/take_first.py
index 547e0f4..cdad243 100644
--- a/janitor/functions/take_first.py
+++ b/janitor/functions/take_first.py
@@ -1,3 +1,4 @@
+"""Implementation of take_first function."""
 from typing import Hashable, Iterable, Union
 import pandas_flavor as pf
 import pandas as pd
@@ -13,20 +14,21 @@ def take_first(
     """
     Take the first row within each group specified by `subset`.
 
-    This method does not mutate the original DataFrame.
-
-    ```python
-        import pandas as pd
-        import janitor
-
-        data = {
-            "a": ["x", "x", "y", "y"],
-            "b": [0, 1, 2, 3]
-        }
-        df = pd.DataFrame(data)
-
-        df.take_first(subset="a", by="b")
-    ```
+    Example:
+
+        >>> import pandas as pd
+        >>> import janitor
+        >>> df = pd.DataFrame({"a": ["x", "x", "y", "y"], "b": [0, 1, 2, 3]})
+        >>> df
+           a  b
+        0  x  0
+        1  x  1
+        2  y  2
+        3  y  3
+        >>> df.take_first(subset="a", by="b")
+           a  b
+        0  x  0
+        2  y  2
 
     :param df: A pandas DataFrame.
     :param subset: Column(s) defining the group.
diff --git a/janitor/io.py b/janitor/io.py
index 918581b..d81456d 100644
--- a/janitor/io.py
+++ b/janitor/io.py
@@ -1,11 +1,13 @@
 import os
+import subprocess
 from glob import glob
+from io import StringIO
 from typing import Iterable, Union
 
 import pandas as pd
 
 from .errors import JanitorError
-from .utils import deprecated_alias
+from .utils import deprecated_alias, check
 
 
 @deprecated_alias(seperate_df="separate_df", filespath="files_path")
@@ -63,9 +65,48 @@ def read_csvs(
                     "Files cannot be concatenated"
                 )
         return pd.concat(
-            list(dfs_dict.values()),  # noqa: PD011
+            list(dfs_dict.values()),
             ignore_index=True,
-            sort=False,
+            sort=False,  # noqa: PD011
         )
     else:
         return dfs_dict
+
+
+def read_commandline(cmd: str, **kwargs) -> pd.DataFrame:
+    """
+    Read a CSV file based on a command-line command.
+
+    For example, you may wish to run the following command on `sep-quarter.csv`
+    before reading it into a pandas DataFrame:
+
+    ```bash
+    cat sep-quarter.csv | grep .SEA1AA
+    ```
+
+    In this case, you can use the following Python code to load the dataframe:
+
+    ```python
+    import janitor as jn
+    df = jn.io.read_commandline("cat data.csv | grep .SEA1AA")
+
+    This function assumes that your command line command will return
+    an output that is parsable using pandas.read_csv and StringIO.
+    We default to using pd.read_csv underneath the hood.
+    Keyword arguments are passed through to read_csv.
+    ```
+
+    :param cmd: Shell command to preprocess a file on disk.
+    :param kwargs: Keyword arguments that are passed through to pd.read_csv().
+    :raises JanitorError: If commandline command is malformed or invalid.
+    :returns: A pandas DataFrame parsed from the stdout of the underlying
+        shell.
+    """
+
+    check("cmd", cmd, [str])
+    outcome = subprocess.run(cmd, shell=True, capture_output=True, text=True)
+    if outcome.returncode != 0:
+        raise JanitorError(outcome.stderr)
+    else:
+        outcome = outcome.stdout
+    return pd.read_csv(StringIO(outcome), **kwargs)


Test patch:
diff --git a/tests/io/test_read_commandline.py b/tests/io/test_read_commandline.py
new file mode 100644
index 0000000..0510e00
--- /dev/null
+++ b/tests/io/test_read_commandline.py
@@ -0,0 +1,63 @@
+import os
+
+import pandas as pd
+import pytest
+
+import janitor.io
+from janitor.errors import JanitorError
+
+
+def test_read_commandline(dataframe):
+    """
+    Test asserts that the dataframe made
+        from the read_commandline function is
+        identical to the test dataframe from
+        which the .csv file was created.
+
+    """
+    # create a temporary .csv file from test data
+    dataframe.to_csv("/tmp/dataframe.csv", index=0)
+
+    # create a new dataframe from the temporary .csv using
+    #   the cat command from the bash commandline
+    df = janitor.io.read_commandline("cat /tmp/dataframe.csv")
+
+    # Make assertion that new dataframe created with read_commandline
+    #   is equal to the test dataframe
+    assert df.equals(dataframe)
+
+    # clean up after the test
+    os.unlink("/tmp/dataframe.csv")
+
+
+def test_read_commandline_bad_cmd(dataframe):
+    """
+    Test 1 raises a TypeError if read_commandline
+        is given an input that is not a string.
+
+    Test 2 raises a JanitorError if
+        read_commandline is given a string
+        which is not a valid bash command.
+
+    Test 3 raises an EmptyDataError if
+        read_commandlind is given a string which
+        is a valid bash command, however results
+        in the shell not creating a dataframe.
+    """
+    # create a temporary .csv file
+    dataframe.to_csv("/tmp/dataframe.csv")
+
+    # Test 1
+    with pytest.raises(TypeError):
+        janitor.io.read_commandline(6)
+
+    # Test 2
+    with pytest.raises(JanitorError):
+        janitor.io.read_commandline("bad command")
+
+    # Test 3
+    with pytest.raises(pd.errors.EmptyDataError):
+        janitor.io.read_commandline("cat")
+
+    # clean up after the tests
+    os.unlink("/tmp/dataframe.csv")
diff --git a/tests/spark/functions/test_clean_names_spark.py b/tests/spark/functions/test_clean_names_spark.py
index 36dc3c2..265aab4 100644
--- a/tests/spark/functions/test_clean_names_spark.py
+++ b/tests/spark/functions/test_clean_names_spark.py
@@ -10,8 +10,10 @@ else:
 import janitor.spark  # noqa: F401 isort:skip
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 def test_clean_names_method_chain(spark_df):
+    """Tests clean names function in a method chain call."""
     spark_df = spark_df.clean_names()
     expected_columns = [
         "a",
@@ -23,8 +25,10 @@ def test_clean_names_method_chain(spark_df):
     assert set(spark_df.columns) == set(expected_columns)
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 def test_clean_names_special_characters(spark_df):
+    """Tests that remove_special parameter takes effect."""
     spark_df = spark_df.clean_names(remove_special=True)
     expected_columns = [
         "a",
@@ -36,8 +40,10 @@ def test_clean_names_special_characters(spark_df):
     assert set(spark_df.columns) == set(expected_columns)
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 def test_clean_names_case_type_uppercase(spark_df):
+    """Tests case_type parameter 'upper' takes effect."""
     spark_df = spark_df.clean_names(case_type="upper")
     expected_columns = [
         "A",
@@ -49,8 +55,10 @@ def test_clean_names_case_type_uppercase(spark_df):
     assert set(spark_df.columns) == set(expected_columns)
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 def test_clean_names_case_type_preserve(spark_df):
+    """Tests case_type parameter 'preserve' takes effect."""
     spark_df = spark_df.clean_names(case_type="preserve")
     expected_columns = [
         "a",
@@ -62,25 +70,31 @@ def test_clean_names_case_type_preserve(spark_df):
     assert set(spark_df.columns) == set(expected_columns)
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 def test_clean_names_case_type_invalid(spark_df):
+    """Tests case_type parameter when it is invalid."""
     with pytest.raises(JanitorError, match=r"case_type must be one of:"):
         spark_df = spark_df.clean_names(case_type="foo")
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 def test_clean_names_camelcase_to_snake(spark_df):
+    """Tests case_type parameter 'snake' takes effect."""
     spark_df = spark_df.selectExpr("a AS AColumnName").clean_names(
         case_type="snake"
     )
     assert list(spark_df.columns) == ["a_column_name"]
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 @pytest.mark.parametrize(
     "strip_underscores", ["both", True, "right", "r", "left", "l"]
 )
 def test_clean_names_strip_underscores(spark_df, strip_underscores):
+    """Tests strip_underscores parameter takes effect."""
     if strip_underscores in ["right", "r"]:
         spark_df = spark_df.selectExpr(
             *[f"`{col}` AS `{col}_`" for col in spark_df.columns]
diff --git a/tests/spark/functions/test_update_where_spark.py b/tests/spark/functions/test_update_where_spark.py
index 9c2acec..aec5e74 100644
--- a/tests/spark/functions/test_update_where_spark.py
+++ b/tests/spark/functions/test_update_where_spark.py
@@ -9,6 +9,7 @@ else:
 import janitor.spark  # noqa: F401 isort:skip
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 def test_update_where_string(dataframe, spark_dataframe):
     """Test update_where and update with a string."""
@@ -29,6 +30,7 @@ def test_update_where_string(dataframe, spark_dataframe):
     )
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 def test_update_where_float(dataframe, spark_dataframe):
     """Test update_where and update with a float."""
@@ -49,6 +51,7 @@ def test_update_where_float(dataframe, spark_dataframe):
     )
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 def test_update_where_int(dataframe, spark_dataframe):
     """Test update_where and update with a int."""
@@ -69,6 +72,7 @@ def test_update_where_int(dataframe, spark_dataframe):
     )
 
 
+@pytest.mark.xfail(reason="causing issues in CI, to be fixed later")
 @pytest.mark.spark_functions
 def test_update_where_column_dne(dataframe, spark_dataframe):
     """Test update_where. Target column name does not exists."""