Repository: ydataai/ydata-profiling

Issue:
ValueError: shape mismatch: value array of shape (136,) could not be broadcast to indexing result of shape (135,)
Hi,
     I have a dataset with 42 numerical features and 143 categorical features. I am on pandas 1.1.0 and pandas-profiling 2.8.0. I did try to use pandas-profiling with the given below commands:

profile=pandas_profiling.ProfileReport(X_train_After_FS_1)
profile.to_file("my_filename.html")

**Error Details :**
HBox(children=(FloatProgress(value=0.0, description='Summarize dataset', max=200.0, style=ProgressStyle(descri…

Traceback (most recent call last):

  File "<ipython-input-13-d112e0e4de59>", line 2, in <module>
    profile.to_file("C:\\Users\\ctystq\\Surajit_Personal\\AMPLifeModels\\Transformation_Project\\LS_Lapse_Propensity\\2.Category-2_RS _Ord_Models\\M1-ANVRSY_MTHS_NON_TR\\EDA_Output\\Plots\\Dataset_Profile_Raw_Data_From_Hive.html")

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas_profiling\profile_report.py", line 245, in to_file
    data = self.to_html()

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas_profiling\profile_report.py", line 348, in to_html
    return self.html

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas_profiling\profile_report.py", line 168, in html
    self._html = self._render_html()

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas_profiling\profile_report.py", line 275, in _render_html
    report = self.report

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas_profiling\profile_report.py", line 162, in report
    self._report = get_report_structure(self.description_set)

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas_profiling\profile_report.py", line 143, in description_set
    self._description_set = describe_df(self.title, self.df)

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas_profiling\model\describe.py", line 63, in describe
    series_description = get_series_descriptions(df, pbar)

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas_profiling\model\summary.py", line 473, in get_series_descriptions
    executor.imap_unordered(multiprocess_1d, args)

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\multiprocessing\pool.py", line 774, in next
    raise value

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\multiprocessing\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas_profiling\model\summary.py", line 450, in multiprocess_1d
    return column, describe_1d(series)

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas_profiling\model\summary.py", line 393, in describe_1d
    series.fillna(np.nan, inplace=True)

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas\core\series.py", line 4523, in fillna
    downcast=downcast,

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas\core\generic.py", line 6079, in fillna
    return self._update_inplace(result)

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas\core\generic.py", line 3955, in _update_inplace
    self._maybe_update_cacher(verify_is_copy=verify_is_copy)

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas\core\generic.py", line 3235, in _maybe_update_cacher
    ref._maybe_cache_changed(cacher[0], self)

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas\core\generic.py", line 3196, in _maybe_cache_changed
    self._mgr.iset(loc, value)

  File "D:\Installed-sw\Anaconda3\envs\root_clone-Hive\lib\site-packages\pandas\core\internals\managers.py", line 1076, in iset
    self._blklocs[blk.mgr_locs.indexer] = np.arange(len(blk))

ValueError: shape mismatch: value array of shape (136,) could not be broadcast to indexing result of shape (135,)
Thanks
Surajit


Discussion:
Hi, I've encountered the same problem with pandas-profiling==2.9.0rc1 with sensitive=True.
As a workaround it seems setting pool_size = 1 fixes the issue in my case - worth trying.

Cheers

Hi all, thanks for bringing this to our attention. The issue seems to be with `series.fillna(np.nan, inplace=True)` here. Can we isolate the issue? @SurajitTest would you have time to ablate parts of the dataset to find the root cause?

`
Hi! 

Have isolated the error - yes it's likely to be the `series.fillna(np.nan, inplace=True)` line. Specifically, in the multiprocess case (pool_size>1) .fillna(inplace=True) seems to be performing multiple in-place mutations to the underlying DataFrame object through the passed series reference, resulting in some kind of race condition where two of the processes try to write to the DataFrame at the same-ish time and the ValueError then occurs. This is also why changing the pool_size to 1 fixes the issue, and why the error doesn't always occur - you probably need enough data and threads to hit the race condition. I've found a way to reliably replicate the problem below using a large-ish data set.

A simple fix might be to use `series = series.fillna(np.nan)` instead, so that there isn't the mutation side-effect. Will submit a pull request in a bit, but writing a test for this might be hard because there needs to be sufficient data for the race condition to hit and this might slow down testing(?).


Minimal code to replicate bug is 
````
from sklearn import datasets
from pandas_profiling import ProfileReport
import pandas as pd

kdd = datasets.fetch_kddcup99(percent10=True)
df = pd.DataFrame(kdd["data"].copy())
ProfileReport(df)
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/anaconda3/envs/test_pandas/lib/python3.6/site-packages/IPython/core/formatters.py in __call__(self, obj)
    343             method = get_real_method(obj, self.print_method)
    344             if method is not None:
--> 345                 return method()
    346             return None
    347         else:

~/projects/pandas-profiling/src/pandas_profiling/profile_report.py in _repr_html_(self)
    407     def _repr_html_(self):
    408         """The ipython notebook widgets user interface gets called by the jupyter notebook."""
--> 409         self.to_notebook_iframe()
    410 
    411     def __repr__(self):

~/projects/pandas-profiling/src/pandas_profiling/profile_report.py in to_notebook_iframe(self)
    387         with warnings.catch_warnings():
    388             warnings.simplefilter("ignore")
--> 389             display(get_notebook_iframe(self))
    390 
    391     def to_widgets(self):

~/projects/pandas-profiling/src/pandas_profiling/report/presentation/flavours/widget/notebook.py in get_notebook_iframe(profile)
     63         output = get_notebook_iframe_src(profile)
     64     elif attribute == "srcdoc":
---> 65         output = get_notebook_iframe_srcdoc(profile)
     66     else:
     67         raise ValueError(

~/projects/pandas-profiling/src/pandas_profiling/report/presentation/flavours/widget/notebook.py in get_notebook_iframe_srcdoc(profile)
     21     width = config["notebook"]["iframe"]["width"].get(str)
     22     height = config["notebook"]["iframe"]["height"].get(str)
---> 23     src = html.escape(profile.to_html())
     24 
     25     iframe = f'<iframe width="{width}" height="{height}" srcdoc="{src}" frameborder="0" allowfullscreen></iframe>'

~/projects/pandas-profiling/src/pandas_profiling/profile_report.py in to_html(self)
    357 
    358         """
--> 359         return self.html
    360 
    361     def to_json(self) -> str:

~/projects/pandas-profiling/src/pandas_profiling/profile_report.py in html(self)
    177     def html(self):
    178         if self._html is None:
--> 179             self._html = self._render_html()
    180         return self._html
    181 

~/projects/pandas-profiling/src/pandas_profiling/profile_report.py in _render_html(self)
    284         from pandas_profiling.report.presentation.flavours import HTMLReport
    285 
--> 286         report = self.report
    287 
    288         disable_progress_bar = not config["progress_bar"].get(bool)

~/projects/pandas-profiling/src/pandas_profiling/profile_report.py in report(self)
    171     def report(self):
    172         if self._report is None:
--> 173             self._report = get_report_structure(self.description_set)
    174         return self._report
    175 

~/projects/pandas-profiling/src/pandas_profiling/profile_report.py in description_set(self)
    152     def description_set(self):
    153         if self._description_set is None:
--> 154             self._description_set = describe_df(self.title, self.df, self._sample)
    155         return self._description_set
    156 

~/projects/pandas-profiling/src/pandas_profiling/model/describe.py in describe(title, df, sample)
     64         total=number_of_tasks, desc="Summarize dataset", disable=disable_progress_bar
     65     ) as pbar:
---> 66         series_description = get_series_descriptions(df, pbar)
     67 
     68         pbar.set_postfix_str("Get variable types")

~/projects/pandas-profiling/src/pandas_profiling/model/summary.py in get_series_descriptions(df, pbar)
    510         with multiprocessing.pool.ThreadPool(pool_size) as executor:
    511             for i, (column, description) in enumerate(
--> 512                 executor.imap_unordered(multiprocess_1d, args)
    513             ):
    514                 pbar.set_postfix_str(f"Describe variable:{column}")

~/anaconda3/envs/test_pandas/lib/python3.6/multiprocessing/pool.py in next(self, timeout)
    733         if success:
    734             return value
--> 735         raise value
    736 
    737     __next__ = next                    # XXX

~/anaconda3/envs/test_pandas/lib/python3.6/multiprocessing/pool.py in worker(inqueue, outqueue, initializer, initargs, maxtasks, wrap_exception)
    117         job, i, func, args, kwds = task
    118         try:
--> 119             result = (True, func(*args, **kwds))
    120         except Exception as e:
    121             if wrap_exception and func is not _helper_reraises_exception:

~/projects/pandas-profiling/src/pandas_profiling/model/summary.py in multiprocess_1d(args)
    487         """
    488         column, series = args
--> 489         return column, describe_1d(series)
    490 
    491     # Multiprocessing of Describe 1D for each column

~/projects/pandas-profiling/src/pandas_profiling/model/summary.py in describe_1d(series)
    430     #print("using series replacement")
    431     #series = series.fillna(np.nan)
--> 432     series.fillna(np.nan,inplace=True)
    433 
    434     # Infer variable types

~/anaconda3/envs/test_pandas/lib/python3.6/site-packages/pandas/core/series.py in fillna(self, value, method, axis, inplace, limit, downcast)
   4521             inplace=inplace,
   4522             limit=limit,
-> 4523             downcast=downcast,
   4524         )
   4525 

~/anaconda3/envs/test_pandas/lib/python3.6/site-packages/pandas/core/generic.py in fillna(self, value, method, axis, inplace, limit, downcast)
   6077         result = self._constructor(new_data)
   6078         if inplace:
-> 6079             return self._update_inplace(result)
   6080         else:
   6081             return result.__finalize__(self, method="fillna")

~/anaconda3/envs/test_pandas/lib/python3.6/site-packages/pandas/core/generic.py in _update_inplace(self, result, verify_is_copy)
   3953         self._clear_item_cache()
   3954         self._mgr = result._mgr
-> 3955         self._maybe_update_cacher(verify_is_copy=verify_is_copy)
   3956 
   3957     def add_prefix(self: FrameOrSeries, prefix: str) -> FrameOrSeries:

~/anaconda3/envs/test_pandas/lib/python3.6/site-packages/pandas/core/generic.py in _maybe_update_cacher(self, clear, verify_is_copy)
   3233                 if len(self) == len(ref):
   3234                     # otherwise, either self or ref has swapped in new arrays
-> 3235                     ref._maybe_cache_changed(cacher[0], self)
   3236 
   3237         if verify_is_copy:

~/anaconda3/envs/test_pandas/lib/python3.6/site-packages/pandas/core/generic.py in _maybe_cache_changed(self, item, value)
   3194         """
   3195         loc = self._info_axis.get_loc(item)
-> 3196         self._mgr.iset(loc, value)
   3197 
   3198     @property

~/anaconda3/envs/test_pandas/lib/python3.6/site-packages/pandas/core/internals/managers.py in iset(self, loc, value)
   1074                 else:
   1075                     blk.delete(blk_locs)
-> 1076                     self._blklocs[blk.mgr_locs.indexer] = np.arange(len(blk))
   1077 
   1078         if len(removed_blknos):

ValueError: shape mismatch: value array of shape (40,) could not be broadcast to indexing result of shape (39,)


> 
Closing PR till tests are included and fixed, will reopen again when done

Resolving patch:
diff --git a/src/pandas_profiling/model/summary.py b/src/pandas_profiling/model/summary.py
--- a/src/pandas_profiling/model/summary.py
+++ b/src/pandas_profiling/model/summary.py
@@ -427,7 +427,7 @@ def describe_boolean_1d(series: pd.Series, series_description: dict) -> dict:
         return stats
 
     # Make sure pd.NA is not in the series
-    series.fillna(np.nan, inplace=True)
+    series = series.fillna(np.nan)
 
     # Infer variable types
     # TODO: use visions for type inference
@@ -679,6 +679,10 @@ def get_scatter_matrix(df, variables):
             targets = continuous_variables
 
         scatter_matrix = {x: {y: "" for y in continuous_variables} for x in targets}
+
+        #check if any na still exists, and remove it before computing scatter matrix
+        df = df.dropna(subset=continuous_variables)
+
         for x in targets:
             for y in continuous_variables:
                 if x in continuous_variables:


Test patch:
diff --git a/tests/issues/test_issue537.py b/tests/issues/test_issue537.py
new file mode 100644
--- /dev/null
+++ b/tests/issues/test_issue537.py
@@ -0,0 +1,126 @@
+"""
+Test for issue 537:
+https://github.com/pandas-profiling/pandas-profiling/issues/537
+
+ValueError: shape mismatch: value array of shape (136,) could not be broadcast to indexing result of shape (135,)
+
+Problem :
+ValueError is raised when running ProfileReport on large datasets and with multiprocessing on (pool_size >1).
+This is likely due to the series.fillna(np.nan, inplace=True) in summary.py seems to be performing multiple in-place
+mutations to the underlying DataFrame object through the passed series reference, resulting in some kind of race
+condition where two of the processes try to write to the DataFrame at the same time and the ValueError then occurs.
+ This is also why changing the pool_size to 1 fixes the issue, and why the error doesn't always occur -
+ you probably need enough data and threads to hit the race condition.
+
+Solution :
+Replace series.fillna(np.nan, inplace=True) with series = series.fillna(np.nan) , negating any side effects from mutation.
+
+
+"""
+
+import multiprocessing
+from gzip import decompress
+from typing import Tuple
+
+import numpy as np
+import pandas as pd
+import requests
+
+from pandas_profiling.model.summary import describe_1d
+
+
+def mock_multiprocess_1d(args) -> Tuple[str, dict]:
+    """Wrapper to process series in parallel.
+        copy of multiprocess_1d function in get_series_descriptions, summary.py
+
+    Args:
+        column: The name of the column.
+        series: The series values.
+
+    Returns:
+        A tuple with column and the series description.
+    """
+    column, series = args
+    return column, describe_1d(series)
+
+
+def test_multiprocessing_describe1d():
+    """
+    this test serves to get a large dataset, and ensure that even across parallelised describe1d operations,
+    there is no ValueError raised. Previously, series.fillna(np.nan,inplace=True) was used instead of
+    series = series.fillna(np.nan) in model.summmary.describe1d, resulting in a race condition where the underlying
+    df was being mutated by two threads at the same time creating a ValueError. This test checks that this does not
+    occur again by running a parallelised describe1d and testing if a ValueError is raised.
+
+    """
+
+    def download_and_process_data():
+        response = requests.get("https://ndownloader.figshare.com/files/5976042")
+        assert response.status_code == 200
+        file = decompress(response.content)
+        text = file.decode()
+        split_text = [i.split(",") for i in filter(lambda x: x, text.split("\n"))]
+        dt = [
+            ("duration", int),
+            ("protocol_type", "S4"),
+            ("service", "S11"),
+            ("flag", "S6"),
+            ("src_bytes", int),
+            ("dst_bytes", int),
+            ("land", int),
+            ("wrong_fragment", int),
+            ("urgent", int),
+            ("hot", int),
+            ("num_failed_logins", int),
+            ("logged_in", int),
+            ("num_compromised", int),
+            ("root_shell", int),
+            ("su_attempted", int),
+            ("num_root", int),
+            ("num_file_creations", int),
+            ("num_shells", int),
+            ("num_access_files", int),
+            ("num_outbound_cmds", int),
+            ("is_host_login", int),
+            ("is_guest_login", int),
+            ("count", int),
+            ("srv_count", int),
+            ("serror_rate", float),
+            ("srv_serror_rate", float),
+            ("rerror_rate", float),
+            ("srv_rerror_rate", float),
+            ("same_srv_rate", float),
+            ("diff_srv_rate", float),
+            ("srv_diff_host_rate", float),
+            ("dst_host_count", int),
+            ("dst_host_srv_count", int),
+            ("dst_host_same_srv_rate", float),
+            ("dst_host_diff_srv_rate", float),
+            ("dst_host_same_src_port_rate", float),
+            ("dst_host_srv_diff_host_rate", float),
+            ("dst_host_serror_rate", float),
+            ("dst_host_srv_serror_rate", float),
+            ("dst_host_rerror_rate", float),
+            ("dst_host_srv_rerror_rate", float),
+            ("labels", "S16"),
+        ]
+        DT = np.dtype(dt)
+        split_text = np.asarray(split_text, dtype=object)
+        for j in range(42):
+            split_text[:, j] = split_text[:, j].astype(DT[j])
+        df = pd.DataFrame(split_text)
+        return df
+
+    def run_multiprocess(df):
+        pool = multiprocessing.pool.ThreadPool(10)
+        args = [(column, series) for column, series in df.iteritems()]
+        results = pool.imap_unordered(mock_multiprocess_1d, args)
+        pool.close()
+        pool.join()
+        list(results)
+
+    try:
+        df = download_and_process_data()
+        run_multiprocess(df)
+    except ValueError:
+        raise Exception("myFunc() raised ValueError unexpectedly!")