Repository: alteryx/woodwork
Pull request #1485 (2022-08-04 00:23:24)

concat_columns handles DataFrames with differing rows
- When `woodwork.concat_columns` is called on two pandas DataFrames with different numbers of rows or mismatched indices, this causes the underlying library, be it pandas, dask or spark to insert null values into the data where there is no feature support.  This can cause the schema to be invalidated, requiring re-inference.

CLAassistant: [![CLA assistant check](https://cla-assistant.io/pull/badge/not_signed)](https://cla-assistant.io/alteryx/woodwork?pullRequest=1485) <br/>Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our [Contributor License Agreement](https://cla-assistant.io/alteryx/woodwork?pullRequest=1485) before we can accept your contribution.<br/><sub>You have signed the CLA already but the status is still pending? Let us [recheck](https://cla-assistant.io/check/alteryx/woodwork?pullRequest=1485) it.</sub>
codecov[bot]: # [Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1485?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) Report
> Merging [#1485](https://codecov.io/gh/alteryx/woodwork/pull/1485?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (d425ef9) into [main](https://codecov.io/gh/alteryx/woodwork/commit/4c27193276f8ee57aa9769c184375d4440f2ea28?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (4c27193) will **decrease** coverage by `80.32%`.
> The diff coverage is `0.00%`.

> :exclamation: Current head d425ef9 differs from pull request most recent head 8e32eca. Consider uploading reports for the commit 8e32eca to get more accurate results

```diff
@@             Coverage Diff             @@
##             main    #1485       +/-   ##
===========================================
- Coverage   98.70%   18.37%   -80.33%     
===========================================
  Files          95       95               
  Lines       10695    10705       +10     
===========================================
- Hits        10556     1967     -8589     
- Misses        139     8738     +8599     
```


| [Impacted Files](https://codecov.io/gh/alteryx/woodwork/pull/1485?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | Coverage Δ | |
|---|---|---|
| [woodwork/tests/utils/test\_concat.py](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdXRpbHMvdGVzdF9jb25jYXQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/utils.py](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdXRpbHMucHk=) | `32.20% <0.00%> (-67.80%)` | :arrow_down: |
| [woodwork/tests/test\_config.py](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdGVzdF9jb25maWcucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/test\_version.py](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdGVzdF92ZXJzaW9uLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/utils/test\_utils.py](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdXRpbHMvdGVzdF91dGlscy5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/type\_system/conftest.py](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdHlwZV9zeXN0ZW0vY29uZnRlc3QucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/utils/test\_read\_file.py](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdXRpbHMvdGVzdF9yZWFkX2ZpbGUucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/accessor/test\_indexers.py](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvYWNjZXNzb3IvdGVzdF9pbmRleGVycy5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/demo\_tests/test\_retail.py](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvZGVtb190ZXN0cy90ZXN0X3JldGFpbC5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/fixtures/datetime\_freq.py](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvZml4dHVyZXMvZGF0ZXRpbWVfZnJlcS5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [58 more](https://codecov.io/gh/alteryx/woodwork/pull/1485/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx)

Cmancuso: This LGTM, I'd have @jeff-hernandez or @ParthivNaresh give it a once over since they might be more familiar with these changes, but should be good to go. 👍 

Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index af1779e..bab244f 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -7,13 +7,14 @@ Future Release
 ==============
     * Enhancements
     * Fixes
+        * Updated concat_columns to work with dataframes with mismatched indices or different shapes (:pr:`1485`)
     * Changes
     * Documentation Changes
        * Add instructions to add new users to woodwork feedstock (:pr:`1483`)
     * Testing Changes
 
     Thanks to the following people for contributing to this release:
-    :user:`gsheni`
+    :user:`gsheni`, :user:`chukarsten`
 
 v0.17.1 July 29, 2022
 =====================
@@ -22,7 +23,7 @@ v0.17.1 July 29, 2022
 
     Thanks to the following people for contributing to this release:
     :user:`bchen1116`, :user:`gsheni`
-    
+
 v0.17.0 July 14, 2022
 =====================
     .. warning::
diff --git a/woodwork/utils.py b/woodwork/utils.py
index 72ee299..944c355 100644
--- a/woodwork/utils.py
+++ b/woodwork/utils.py
@@ -515,6 +515,21 @@ def concat_columns(objs, validate_schema=True):
 
     combined_df = lib.concat(objs, axis=1, join="outer")
 
+    # The lib.concat breaks the woodwork schema for dataframes with different shapes
+    # or mismatched indices.
+    mask = combined_df.isnull().any()
+    null_cols = mask[mask].index
+    if not ww.accessor_utils._is_dask_dataframe(combined_df):
+        null_cols = null_cols.to_numpy()
+    else:
+        null_cols = list(null_cols)
+    for null_col in null_cols:
+        if null_col in logical_types and isinstance(
+            logical_types[null_col],
+            ww.logical_types.Integer,
+        ):
+            logical_types.pop(null_col)
+
     # Initialize Woodwork with all of the typing information from the input objs
     # performing type inference on any columns that did not already have Woodwork initialized
     combined_df.ww.init(


Test patch:
diff --git a/woodwork/tests/utils/test_concat.py b/woodwork/tests/utils/test_concat.py
index c9d1dc6..47cee1e 100644
--- a/woodwork/tests/utils/test_concat.py
+++ b/woodwork/tests/utils/test_concat.py
@@ -4,8 +4,14 @@ import pandas as pd
 import pytest
 
 import woodwork as ww
-from woodwork.accessor_utils import _is_spark_dataframe
-from woodwork.logical_types import Categorical, Double, Integer
+from woodwork.accessor_utils import _is_dask_dataframe, _is_spark_dataframe
+from woodwork.logical_types import (
+    BooleanNullable,
+    Categorical,
+    Double,
+    Integer,
+    IntegerNullable,
+)
 from woodwork.tests.testing_utils import to_pandas
 from woodwork.utils import concat_columns
 
@@ -397,17 +403,6 @@ def test_concat_cols_validate_schema(mock_validate_accessor_params, sample_df):
 
 
 def test_concat_cols_mismatched_index_adds_single_nan(sample_df):
-    sample_df.ww.init()
-
-    df1 = sample_df.ww.loc[[0, 1, 2], ["id", "full_name"]]
-    df2 = sample_df.ww.loc[[1, 2, 3], ["signup_date", "email"]]
-
-    if isinstance(sample_df, pd.DataFrame):
-        # Only pandas shows the dtype change because Dask and Spark won't show until compute
-        error = "Error converting datatype for id from type float64 to type int64. Please confirm the underlying data is consistent with logical type Integer."
-        with pytest.raises(ww.exceptions.TypeConversionError, match=error):
-            concat_columns([df1, df2])
-
     # If the dtype can handle nans, it won't change
     sample_df.ww.init(logical_types={"id": "IntegerNullable"})
 
@@ -540,3 +535,48 @@ def test_concat_with_none(sample_df):
     error = "'NoneType' object has no attribute 'ww'"
     with pytest.raises(AttributeError, match=error):
         concat_columns([df1, df2, None])
+
+
+@pytest.mark.parametrize(
+    "nullable_type",
+    [
+        "integer",
+        pytest.param(
+            "boolean",
+            marks=pytest.mark.xfail(reason="Bug with BooleanNullable inference."),
+        ),
+    ],
+)
+def test_concat_shorter_null_int(sample_df, nullable_type):
+    """Tests that concatenating dataframes with different numbers of rows works."""
+    sample_df.ww.init()
+    df1 = sample_df.ww[
+        [
+            "email",
+            "phone_number",
+        ]
+    ]
+
+    # Purposefully create a dataframe with a non-nullable integer column
+    # that's shorter than the one to concat with
+    if _is_dask_dataframe(sample_df):
+        pytest.skip(
+            "Slicing dataframe with respect to rows is not supported with Dask input",
+        )
+    df2 = sample_df.ww[
+        [
+            nullable_type,
+            "categorical",
+        ]
+    ].iloc[1:, :]
+    df2.ww.init()
+
+    result = concat_columns([df1, df2])
+    expected_type = IntegerNullable if nullable_type == "integer" else BooleanNullable
+    assert isinstance(result.ww.logical_types[nullable_type], expected_type)
+    assert (
+        result.shape[0] == df1.shape[0]
+    ), "The output of concat_columns has the incorrect number of rows."
+    assert (
+        result.shape[1] == df1.shape[1] + df2.shape[1]
+    ), "The output of concat_columns has the incorrect number of columns."