Repository: modin-project/modin

Issue:
BUG: ray remote error causes flood of disruptive error messages in interactive python
### Modin version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest released version of Modin.

- [X] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow [this guide](https://modin.readthedocs.io/en/stable/getting_started/installation.html#installing-from-the-github-master-branch).)


### Reproducible Example

```python
import modin.pandas as pd
from modin.config import NPartitions, MinPartitionSize
import numpy as np

NPartitions.put(16)
MinPartitionSize.put(32)
df = pd.DataFrame(np.zeros((10_000,10_000)))
def f(x):
  raise RuntimeError()
print(df.applymap(f))
```


### Issue Description

When a ray task raises an error in an ipython session, I get several "Unhandled error" messages in my session output after some delay (about 1 second). The delay is also annoying because sometimes messages appear while I'm typing my next command. If I initialize ray with `log_to_driver=False` I still get the error. Setting environment variable `RAY_IGNORE_UNHANDLED_ERRORS=1` does get rid of the spam.

See root cause issue #28216.

### Expected Behavior

I would like to only see the nicer main thread error, not any of the 800 messages from failed ray tasks.

### Error Logs

Spammed ray message that appears about 800 times, at least once before the main thread error and several times after:

<details>

```python-traceback

2022-09-21 12:06:08,178	ERROR worker.py:399 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::_apply_func() (pid=36085, ip=127.0.0.1)
  File "/Users/maheshvashishtha/software_sources/modin/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py", line 410, in _apply_func
    result = func(partition, *args, **kwargs)
  File "/Users/maheshvashishtha/software_sources/modin/modin/core/dataframe/algebra/map.py", line 48, in <lambda>
    lambda x: function(x, *args, **kwargs), *call_args, **call_kwds
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/frame.py", line 8933, in applymap
    return self.apply(infer).__finalize__(self, "applymap")
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/frame.py", line 8848, in apply
    return op.apply().__finalize__(self, method="apply")
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/apply.py", line 733, in apply
    return self.apply_standard()
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/apply.py", line 857, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/apply.py", line 873, in apply_series_generator
    results[i] = self.f(v)
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/frame.py", line 8931, in infer
    return lib.map_infer(x.astype(object)._values, func, ignore_na=ignore_na)
  File "pandas/_libs/lib.pyx", line 2870, in pandas._libs.lib.map_infer
  File "<ipython-input-1-3c6216105afe>", line 9, in f
RuntimeError

```

</details>

Nicer Main thread error:

<details>

```python-traceback
---------------------------------------------------------------------------
RayTaskError(RuntimeError)                Traceback (most recent call last)
Input In [1], in <cell line: 10>()
      8 def f(x):
      9   raise RuntimeError()
---> 10 print(df.applymap(f))

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/pandas/base.py:3371, in BasePandasDataset.__str__(self)
   3363 def __str__(self):  # pragma: no cover
   3364     """
   3365     Return str(self).
   3366
   (...)
   3369     str
   3370     """
-> 3371     return repr(self)

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/pandas/dataframe.py:223, in DataFrame.__repr__(self)
    220         i -= 1
    222     num_cols += len(self.columns) - i
--> 223 result = repr(self._build_repr_df(num_rows, num_cols))
    224 if len(self.index) > num_rows or len(self.columns) > num_cols:
    225     # The split here is so that we don't repr pandas row lengths.
    226     return result.rsplit("\n\n", 1)[0] + "\n\n[{0} rows x {1} columns]".format(
    227         len(self.index), len(self.columns)
    228     )

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/pandas/base.py:198, in BasePandasDataset._build_repr_df(self, num_rows, num_cols)
    196 else:
    197     indexer = row_indexer
--> 198 return self.iloc[indexer]._query_compiler.to_pandas()

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/core/storage_formats/pandas/query_compiler.py:277, in PandasQueryCompiler.to_pandas(self)
    276 def to_pandas(self):
--> 277     return self._modin_frame.to_pandas()

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/core/dataframe/pandas/dataframe/dataframe.py:124, in lazy_metadata_decorator.<locals>.decorator.<locals>.run_f_on_minimally_updated_metadata(self, *args, **kwargs)
    122     elif apply_axis == "rows":
    123         obj._propagate_index_objs(axis=0)
--> 124 result = f(self, *args, **kwargs)
    125 if apply_axis is None and not transpose:
    126     result._deferred_index = self._deferred_index

File ~/software_sources/modin/modin/core/dataframe/pandas/dataframe/dataframe.py:3056, in PandasDataframe.to_pandas(self)
   3047 @lazy_metadata_decorator(apply_axis="both")
   3048 def to_pandas(self):
   3049     """
   3050     Convert this Modin DataFrame to a pandas DataFrame.
   3051
   (...)
   3054     pandas.DataFrame
   3055     """
-> 3056     df = self._partition_mgr_cls.to_pandas(self._partitions)
   3057     if df.empty:
   3058         df = pandas.DataFrame(columns=self.columns, index=self.index)

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/core/dataframe/pandas/partitioning/partition_manager.py:644, in PandasDataframePartitionManager.to_pandas(cls, partitions)
    629 @classmethod
    630 def to_pandas(cls, partitions):
    631     """
    632     Convert NumPy array of PandasDataframePartition to pandas DataFrame.
    633
   (...)
    642         A pandas DataFrame
    643     """
--> 644     retrieved_objects = [[obj.to_pandas() for obj in part] for part in partitions]
    645     if all(
    646         isinstance(part, pandas.Series) for row in retrieved_objects for part in row
    647     ):
    648         axis = 0

File ~/software_sources/modin/modin/core/dataframe/pandas/partitioning/partition_manager.py:644, in <listcomp>(.0)
    629 @classmethod
    630 def to_pandas(cls, partitions):
    631     """
    632     Convert NumPy array of PandasDataframePartition to pandas DataFrame.
    633
   (...)
    642         A pandas DataFrame
    643     """
--> 644     retrieved_objects = [[obj.to_pandas() for obj in part] for part in partitions]
    645     if all(
    646         isinstance(part, pandas.Series) for row in retrieved_objects for part in row
    647     ):
    648         axis = 0

File ~/software_sources/modin/modin/core/dataframe/pandas/partitioning/partition_manager.py:644, in <listcomp>(.0)
    629 @classmethod
    630 def to_pandas(cls, partitions):
    631     """
    632     Convert NumPy array of PandasDataframePartition to pandas DataFrame.
    633
   (...)
    642         A pandas DataFrame
    643     """
--> 644     retrieved_objects = [[obj.to_pandas() for obj in part] for part in partitions]
    645     if all(
    646         isinstance(part, pandas.Series) for row in retrieved_objects for part in row
    647     ):
    648         axis = 0

File ~/software_sources/modin/modin/core/dataframe/pandas/partitioning/partition.py:145, in PandasDataframePartition.to_pandas(self)
    132 def to_pandas(self):
    133     """
    134     Convert the object wrapped by this partition to a ``pandas.DataFrame``.
    135
   (...)
    143     only need to call `get`.
    144     """
--> 145     dataframe = self.get()
    146     assert isinstance(dataframe, (pandas.DataFrame, pandas.Series))
    147     return dataframe

File ~/software_sources/modin/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py:80, in PandasOnRayDataframePartition.get(self)
     78 if len(self.call_queue):
     79     self.drain_call_queue()
---> 80 result = ray.get(self._data)
     81 logger.debug(f"EXIT::Partition.get::{self._identity}")
     82 return result

File ~/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/ray/_private/client_mode_hook.py:105, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    103     if func.__name__ != "init" or is_client_mode_enabled_by_default:
    104         return getattr(ray, func.__name__)(*args, **kwargs)
--> 105 return func(*args, **kwargs)

File ~/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/ray/_private/worker.py:2275, in get(object_refs, timeout)
   2273     worker.core_worker.dump_object_store_memory_usage()
   2274 if isinstance(value, RayTaskError):
-> 2275     raise value.as_instanceof_cause()
   2276 else:
   2277     raise value

RayTaskError(RuntimeError): ray::_apply_func() (pid=36162, ip=127.0.0.1)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::_apply_func() (pid=36150, ip=127.0.0.1)
  File "/Users/maheshvashishtha/software_sources/modin/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py", line 410, in _apply_func
    result = func(partition, *args, **kwargs)
  File "/Users/maheshvashishtha/software_sources/modin/modin/core/dataframe/algebra/map.py", line 48, in <lambda>
    lambda x: function(x, *args, **kwargs), *call_args, **call_kwds
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/frame.py", line 8933, in applymap
    return self.apply(infer).__finalize__(self, "applymap")
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/frame.py", line 8848, in apply
    return op.apply().__finalize__(self, method="apply")
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/apply.py", line 733, in apply
    return self.apply_standard()
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/apply.py", line 857, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/apply.py", line 873, in apply_series_generator
    results[i] = self.f(v)
  File "/Users/maheshvashishtha/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pandas/core/frame.py", line 8931, in infer
    return lib.map_infer(x.astype(object)._values, func, ignore_na=ignore_na)
  File "pandas/_libs/lib.pyx", line 2870, in pandas._libs.lib.map_infer
  File "<ipython-input-1-3c6216105afe>", line 9, in f
RuntimeError
```

</details>


### Installed Versions

<details>

UserWarning: Setuptools is replacing distutils.

INSTALLED VERSIONS
------------------
commit           : 10c24d36d4eff1dfa8ba6a0ce87fe1d0a9753c6b
python           : 3.10.4.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 21.5.0
Version          : Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:22 PDT 2022; root:xnu-8020.121.3~4/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

Modin dependencies
------------------
modin            : 0.15.0+143.g10c24d36d.dirty
ray              : 2.0.0
dask             : 2022.7.1
distributed      : 2022.7.1
omniscidbe       : None

pandas dependencies
-------------------
pandas           : 1.4.4
numpy            : 1.23.2
pytz             : 2022.2.1
dateutil         : 2.8.2
setuptools       : 61.2.0
pip              : 22.2.2
Cython           : None
pytest           : 7.1.2
hypothesis       : None
sphinx           : 5.1.1
blosc            : None
feather          : 0.4.1
xlsxwriter       : None
lxml.etree       : 4.9.1
html5lib         : None
pymysql          : None
psycopg2         : 2.9.3
jinja2           : 3.1.2
IPython          : 8.4.0
pandas_datareader: None
bs4              : 4.11.1
bottleneck       : None
brotli           : 1.0.9
fastparquet      : 0.8.1
fsspec           : 2022.7.1
gcsfs            : None
markupsafe       : 2.1.1
matplotlib       : 3.5.2
numba            : None
numexpr          : 2.8.3
odfpy            : None
openpyxl         : 3.0.10
pandas_gbq       : 0.17.7
pyarrow          : 8.0.0
pyreadstat       : None
pyxlsb           : None
s3fs             : 2022.7.1
scipy            : 1.9.0
snappy           : None
sqlalchemy       : 1.4.39
tables           : 3.7.0
tabulate         : None
xarray           : 2022.6.0
xlrd             : 2.0.1
xlwt             : None
zstandard        : None

</details>



Discussion:
@modin-project/modin-contributors @modin-project/modin-core @vnlitvinov is it acceptable to fix this issue by setting `os.environ["RAY_IGNORE_UNHANDLED_ERRORS"]="1"` within Modin by default as @scv119 suggests [here](https://github.com/ray-project/ray/issues/28216#issuecomment-1253978996)? I don't feel good about modifying the user's environment variables, but is it okay to do so?

Resolving patch:
diff --git a/modin/core/execution/ray/common/utils.py b/modin/core/execution/ray/common/utils.py
--- a/modin/core/execution/ray/common/utils.py
+++ b/modin/core/execution/ray/common/utils.py
@@ -42,6 +42,8 @@
 # https://github.com/ray-project/ray/blob/4692e8d8023e789120d3f22b41ffb136b50f70ea/python/ray/_private/ray_constants.py#L57-L62
 _MAC_OBJECT_STORE_LIMIT_BYTES = 2 * 2**30
 
+_RAY_IGNORE_UNHANDLED_ERRORS_VAR = "RAY_IGNORE_UNHANDLED_ERRORS"
+
 ObjectIDType = ray.ObjectRef
 if version.parse(ray.__version__) >= version.parse("1.2.0"):
     from ray.util.client.common import ClientObjectRef
@@ -147,6 +149,14 @@ def initialize_ray(
     else:
         NPartitions._put(num_cpus)
 
+    # TODO(https://github.com/ray-project/ray/issues/28216): remove this
+    # workaround once Ray gives a better way to suppress task errors.
+    # Ideally we would not set global environment variables.
+    # If user has explicitly set _RAY_IGNORE_UNHANDLED_ERRORS_VAR, don't
+    # don't override its value.
+    if _RAY_IGNORE_UNHANDLED_ERRORS_VAR not in os.environ:
+        os.environ[_RAY_IGNORE_UNHANDLED_ERRORS_VAR] = "1"
+
 
 def _get_object_store_memory() -> Optional[int]:
     """


Test patch: