Repository: modin-project/modin

Issue:
UserWarning: Distributing <class 'NoneType'> object. This may take some time.
I think that we should not give this warning if the data that will be placed in the distribution storage is small in size.


Discussion:
Hello, I'm interested in doing this as my first issue. Would that be okay? I'm assuming I would just need to add a size check in [dataframe.py](https://github.com/modin-project/modin/blob/main/modin/pandas/dataframe.py) and [series.py](https://github.com/modin-project/modin/blob/main/modin/pandas/series.py) where the warning is, is that correct?. Also, what would be a good size to check for?
Hi @Jayson729! It will be great!

> I'm assuming I would just need to add a size check in [dataframe.py](https://github.com/modin-project/modin/blob/main/modin/pandas/dataframe.py) and [series.py](https://github.com/modin-project/modin/blob/main/modin/pandas/series.py) where the warning is, is that correct?

Correct.

> Also, what would be a good size to check for?

We can assume that if constructing a pandas dataframe took half a second, then placing it into distributed storage will not take so long to inform the user about it. In this case, we will not need to also check the correlation of the execution time of `from_pandas` function from the shape of the dataframe and its types.

That is, the creation of a warning should be placed between the creation of a pandas dataframe and the call of `from_pandas` function. And this should only be done when the creation of a dataframe takes more than half a second. For this, you can use the function `time` (`from time import time`).

**Note:** `half a second` is an empirical value that we can change if necessary, but it reflects my current experience.
Hey @anmyachev,

I just want to make sure I understand everything correctly. Your idea for timing is to put the check in this location ([dataframe.py#L255](https://github.com/modin-project/modin/blob/002125b55b0d75ef3e025c9d540b2b9005840db5/modin/pandas/dataframe.py#L255)), correct?

If yes, then from some testing it seems like the length of time for creating the dataframe varies pretty wildly depending on the datatype of the data passed. Numpy ndarray's seem to take longer on converting a Pandas DataFrame to a Modin DataFrame, while python lists take longer actually creating the Pandas DataFrame. 

This is what I used to test this (at [dataframe.py#L255](https://github.com/modin-project/modin/blob/002125b55b0d75ef3e025c9d540b2b9005840db5/modin/pandas/dataframe.py#L255)):

```python
start = time()
pandas_df = pandas.DataFrame(
    data=data, index=index, columns=columns, dtype=dtype, copy=copy
)
print(f'Time creating DataFrame: {time() - start}')

start = time()
self._query_compiler = from_pandas(pandas_df)._query_compiler
print(f'Time converting Pandas DataFrame to Modin: {time() - start}')
```

and then in another file this is how the DataFrames were created, with some other tests not included in this:

```python
print('30000 x 30000 ndarray')
pd.DataFrame(np.random.rand(30000, 30000))

print('10000 x 10000 python list')
pd.DataFrame([[randint(-10000, 10000) for _ in range(10000)] for _ in range(10000)])
```

and then here's what was printed:

> 30000 x 30000 ndarray
> Time creating DataFrame: 0.00010061264038085938
> Time converting Pandas DataFrame to Modin: 28.473838806152344

> 10000 x 10000 python list
> Time creating DataFrame: 19.381755113601685
> Time converting Pandas DataFrame to Modin: 0.745363712310791

Due to this I think this might not be a good metric to use for size, is my thought process correct? If so, what would be a better metric for checking size? If not, I'd appreciate you letting me know what my misunderstanding is. Thanks a lot for your help!
Thanks for the numbers! It's clear that there is no reliable connection this way now.

Let's now try to use the shape from created pandas DataFrame without paying attention to the types (even those that do not support zero copying). Maybe df.shape < (10_000, 16) at the start point. We can check cases where almost all data is numeric and the edge case where all columns are of string type (strings of reasonable length, no more than 250 characters, for example).
Doesn't this run into a problem if the creating the DataFrame itself takes a significant amount of time (such as the python list example from my last message)? I also did some tests just using sys.getsizeof() on the data before it's passed to the DataFrame, but that also seemed fairly unreliable. Currently I'm thinking we either need to do different checks for size based the datatype or maybe somehow detect the running time of creating the pandas DataFrame, maybe with another program thread? Am I mistaken in some way or do you have any other ideas?
> Doesn't this run into a problem if the creating the DataFrame itself takes a significant amount of time (such as the python list example from my last message)?

This is definitely not a problem, since this is the time of the pandas constructor and users already have some expectations regarding the time of its work in different cases. So my view on this problem is that we should only give a message when the additional part (putting it into distributed storage) takes up a significant part.
That makes complete sense, thanks for explaining. 

I tested with both df.shape and df.size. `df.size > 100_000_000` seems to be a good metric on my computer (about 1 second regardless of data type). The growth also seems to be linear, 100_000_000 takes about 1 second and 400_000_000 takes about 4.

I also did some testing in series.py, but distributing here rarely took a significant amount of time (500_000_000 size was about 1 second). 

I figure we could use `df.size < 25_000_000` for dataframe.py and `df.size < 100_000_000` in series.py to take into account slower computers. Do these bounds seem reasonable?
> I figure we could use df.size < 25_000_000 for dataframe.py and df.size < 100_000_000 in series.py to take into account slower computers. Do these bounds seem reasonable?

At first glance, it looks good!
Just to be sure. How long does `from_pandas` function take you for these cases?

`pd.Series(np.random.randint(1_000_000, size=(100_000_000)), dtype=str)`
and
`pd.DataFrame(np.random.randint(1_000_000, size=(1_000_000, 25)), dtype=str)`
Good catch, these took significantly longer than their integer counterparts. The Series initialization took 17.2 seconds and the DataFrame took 6.2 seconds. A size of 10_000_000 for Series and a size of 5_000_000 for DataFrames both took about a second on my system. So for bounds when dtype=str we could probably use `df.size > 2_500_000` for Series and `df.size > 1_000_000` for DataFrames. Does this also seem reasonable, and are there any other data types you think we should specifically check for?

Also, sorry for the delay on responding, I was away from my computer so I couldn't test.
> So for bounds when dtype=str we could probably use df.size > 2_500_000 for Series and df.size > 1_000_000 for DataFrames. 

Based on the time numbers I received from you (thanks!), I believe I would like to start with these dimensions for all types and see how users like it, and then expand for some types (more for numeric types, as they support faster serialization) if there is positive feedback. 
Sounds good, I'll try to get the pull request in today. Thanks for all your help!

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -205,9 +205,6 @@ def __init__(
                 self._query_compiler = distributed_frame._query_compiler
                 return
 
-            warnings.warn(
-                "Distributing {} object. This may take some time.".format(type(data))
-            )
             if isinstance(data, pandas.Index):
                 pass
             elif (
@@ -253,6 +250,12 @@ def __init__(
             pandas_df = pandas.DataFrame(
                 data=data, index=index, columns=columns, dtype=dtype, copy=copy
             )
+            if pandas_df.size >= 1_000_000:
+                warnings.warn(
+                    "Distributing {} object. This may take some time.".format(
+                        type(data)
+                    )
+                )
             self._query_compiler = from_pandas(pandas_df)._query_compiler
         else:
             self._query_compiler = query_compiler
diff --git a/modin/pandas/series.py b/modin/pandas/series.py
--- a/modin/pandas/series.py
+++ b/modin/pandas/series.py
@@ -137,26 +137,28 @@ def __init__(
                 query_compiler.columns = pandas.Index([MODIN_UNNAMED_SERIES_LABEL])
         if query_compiler is None:
             # Defaulting to pandas
-            warnings.warn(
-                "Distributing {} object. This may take some time.".format(type(data))
-            )
             if name is None:
                 name = MODIN_UNNAMED_SERIES_LABEL
                 if isinstance(data, pandas.Series) and data.name is not None:
                     name = data.name
 
-            query_compiler = from_pandas(
-                pandas.DataFrame(
-                    pandas.Series(
-                        data=data,
-                        index=index,
-                        dtype=dtype,
-                        name=name,
-                        copy=copy,
-                        fastpath=fastpath,
+            pandas_df = pandas.DataFrame(
+                pandas.Series(
+                    data=data,
+                    index=index,
+                    dtype=dtype,
+                    name=name,
+                    copy=copy,
+                    fastpath=fastpath,
+                )
+            )
+            if pandas_df.size >= 2_500_000:
+                warnings.warn(
+                    "Distributing {} object. This may take some time.".format(
+                        type(data)
                     )
                 )
-            )._query_compiler
+            query_compiler = from_pandas(pandas_df)._query_compiler
         self._query_compiler = query_compiler.columnarize()
         if name is not None:
             self.name = name


Test patch:
diff --git a/modin/tests/pandas/dataframe/test_default.py b/modin/tests/pandas/dataframe/test_default.py
--- a/modin/tests/pandas/dataframe/test_default.py
+++ b/modin/tests/pandas/dataframe/test_default.py
@@ -1473,3 +1473,37 @@ def test_df_from_series_with_tuple_name():
     df_equals(pd.DataFrame(pandas.Series(name=("a", 1))), pandas_result)
     # 2. Creating a Modin DF from Modin Series
     df_equals(pd.DataFrame(pd.Series(name=("a", 1))), pandas_result)
+
+
+def test_large_df_warns_distributing_takes_time():
+    # https://github.com/modin-project/modin/issues/6574
+
+    regex = r"Distributing (.*) object\. This may take some time\."
+    with pytest.warns(UserWarning, match=regex):
+        pd.DataFrame(np.random.randint(1_000_000, size=(100_000, 10)))
+
+
+def test_large_series_warns_distributing_takes_time():
+    # https://github.com/modin-project/modin/issues/6574
+
+    regex = r"Distributing (.*) object\. This may take some time\."
+    with pytest.warns(UserWarning, match=regex):
+        pd.Series(np.random.randint(1_000_000, size=(2_500_000)))
+
+
+def test_df_does_not_warn_distributing_takes_time():
+    # https://github.com/modin-project/modin/issues/6574
+
+    regex = r"Distributing (.*) object\. This may take some time\."
+    with warnings.catch_warnings():
+        warnings.filterwarnings("error", regex, UserWarning)
+        pd.DataFrame(np.random.randint(1_000_000, size=(100_000, 9)))
+
+
+def test_series_does_not_warn_distributing_takes_time():
+    # https://github.com/modin-project/modin/issues/6574
+
+    regex = r"Distributing (.*) object\. This may take some time\."
+    with warnings.catch_warnings():
+        warnings.filterwarnings("error", regex, UserWarning)
+        pd.Series(np.random.randint(1_000_000, size=(2_400_000)))
diff --git a/modin/tests/pandas/test_series.py b/modin/tests/pandas/test_series.py
--- a/modin/tests/pandas/test_series.py
+++ b/modin/tests/pandas/test_series.py
@@ -18,6 +18,7 @@
 import json
 import sys
 import unittest.mock as mock
+import warnings
 
 import matplotlib
 import numpy as np
@@ -26,7 +27,7 @@
 import pytest
 from numpy.testing import assert_array_equal
 from pandas.core.indexing import IndexingError
-from pandas.errors import SpecificationError
+from pandas.errors import PerformanceWarning, SpecificationError
 
 import modin.pandas as pd
 from modin.config import Engine, NPartitions, StorageFormat
@@ -3429,13 +3430,10 @@ def test_sub(data):
 
 def test_6782():
     datetime_scalar = datetime.datetime(1970, 1, 1, 0, 0)
-    with pytest.warns(UserWarning) as warns:
-        _ = pd.Series([datetime.datetime(2000, 1, 1)]) - datetime_scalar
-        for warn in warns.list:
-            assert (
-                "Adding/subtracting object-dtype array to DatetimeArray not vectorized"
-                not in str(warn)
-            )
+    match = "Adding/subtracting object-dtype array to DatetimeArray not vectorized"
+    with warnings.catch_warnings():
+        warnings.filterwarnings("error", match, PerformanceWarning)
+        pd.Series([datetime.datetime(2000, 1, 1)]) - datetime_scalar
 
 
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)