Repository: modin-project/modin

Issue:
read_excel prevent from further save of the Excel file while python session is running
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Windows 10 20H2
- **Modin version** (`modin.__version__`): 0.8.2
- **Python version**: Python 3.8.3

An really simple example file to reproduce:
[test.xlsx](https://github.com/modin-project/modin/files/5570144/test.xlsx)

- **Code we can use to reproduce**:
```python
import modin.pandas as pd
df = pd.read_excel('test.xlsx')
```
Then open the Excel file (if not already open), and try to save it.

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

Contrary to behavior in Pandas, using pd.read_excel with modin, prevent any further saving of the Excel files afterwards, until the python session is closed.
While trying to do it in Excel, get multiple error pop-up talking about "sharing issue", given some random number as Excel file then.
For my use case, it's quite handy to get the Excel file and the python session open in the same time, loading the data, working on it with pandas, and potentially (especially during developpement) making small change in Excel and reloading them right away in python. Also, that doesn't seem like a safe behavior and might hide something bigger ? don't know.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

![image](https://user-images.githubusercontent.com/62483870/99733905-31a74580-2ac2-11eb-9124-bbe3da1255cc.png)
![image](https://user-images.githubusercontent.com/62483870/99733925-3a981700-2ac2-11eb-9d70-564b2f7cda4f.png)
![image](https://user-images.githubusercontent.com/62483870/99733969-4a176000-2ac2-11eb-902c-b8a2584f4a13.png)
(sorry it's in french)



Discussion:
Hi @GentilsTo, thanks for posting!

We should make sure that Modin is not hanging on to any open file handles just in case, but I cannot see any cases of this at a glance in the code. It is a perfectly reasonable workflow to do this, and is safe as long as you are not saving a new copy and loading at the same time.

Out of curiosity, what kinds of changes do you make to the excel files? We are prototyping an excel interface that shares the execution, and I'm interested to understand your workflow because it seems like it would fit this interface quite well.

Resolving patch:
diff --git a/modin/config/envvars.py b/modin/config/envvars.py
--- a/modin/config/envvars.py
+++ b/modin/config/envvars.py
@@ -12,6 +12,7 @@
 # governing permissions and limitations under the License.
 
 import os
+import sys
 from textwrap import dedent
 import warnings
 from packaging import version
@@ -208,6 +209,18 @@ class TestDatasetSize(EnvironmentVariable, type=str):
     choices = ("Small", "Normal", "Big")
 
 
+class TrackFileLeaks(EnvironmentVariable, type=bool):
+    """
+    Whether to track for open file handles leakage during testing
+    """
+
+    varname = "MODIN_TEST_TRACK_FILE_LEAKS"
+    # Turn off tracking on Windows by default because
+    # psutil's open_files() can be extremely slow on Windows (up to adding a few hours).
+    # see https://github.com/giampaolo/psutil/pull/597
+    default = sys.platform != "win32"
+
+
 def _check_vars():
     """
     Look out for any environment variables that start with "MODIN_" prefix
diff --git a/modin/engines/base/io/text/excel_dispatcher.py b/modin/engines/base/io/text/excel_dispatcher.py
--- a/modin/engines/base/io/text/excel_dispatcher.py
+++ b/modin/engines/base/io/text/excel_dispatcher.py
@@ -41,7 +41,6 @@ def _read(cls, io, **kwargs):
             return cls.single_worker_read(io, **kwargs)
 
         from zipfile import ZipFile
-        from openpyxl import load_workbook
         from openpyxl.worksheet.worksheet import Worksheet
         from openpyxl.worksheet._reader import WorksheetReader
         from openpyxl.reader.excel import ExcelReader
@@ -59,12 +58,23 @@ def _read(cls, io, **kwargs):
             "Parallel `read_excel` is a new feature! Please email "
             "bug_reports@modin.org if you run into any problems."
         )
-        wb = load_workbook(filename=io, read_only=True)
-        # Get shared strings
-        ex = ExcelReader(io, read_only=True)
-        ex.read_manifest()
-        ex.read_strings()
-        ws = Worksheet(wb)
+
+        # NOTE: ExcelReader() in read-only mode does not close file handle by itself
+        # work around that by passing file object if we received some path
+        io_file = open(io, "rb") if isinstance(io, str) else io
+        try:
+            ex = ExcelReader(io_file, read_only=True)
+            ex.read()
+            wb = ex.wb
+
+            # Get shared strings
+            ex.read_manifest()
+            ex.read_strings()
+            ws = Worksheet(wb)
+        finally:
+            if isinstance(io, str):
+                # close only if it were us who opened the object
+                io_file.close()
 
         with ZipFile(io) as z:
             from io import BytesIO


Test patch:
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -28,6 +28,7 @@
 import csv
 
 from .utils import (
+    check_file_leaks,
     df_equals,
     json_short_string,
     json_short_bytes,
@@ -1184,6 +1185,7 @@ def test_from_clipboard():
 
 
 @pytest.mark.xfail(reason="read_excel is broken for now, see #1733 for details")
+@check_file_leaks
 def test_from_excel():
     setup_excel_file(NROWS)
 
@@ -1195,6 +1197,7 @@ def test_from_excel():
     teardown_excel_file()
 
 
+@check_file_leaks
 def test_from_excel_engine():
     setup_excel_file(NROWS)
 
@@ -1207,6 +1210,7 @@ def test_from_excel_engine():
     teardown_excel_file()
 
 
+@check_file_leaks
 def test_from_excel_index_col():
     setup_excel_file(NROWS)
 
@@ -1219,6 +1223,7 @@ def test_from_excel_index_col():
     teardown_excel_file()
 
 
+@check_file_leaks
 def test_from_excel_all_sheets():
     setup_excel_file(NROWS)
 
@@ -1236,6 +1241,7 @@ def test_from_excel_all_sheets():
     teardown_excel_file()
 
 
+@check_file_leaks
 def test_from_excel_sheetname_title():
     path = "modin/pandas/test/data/excel_sheetname_title.xlsx"
     modin_df = pd.read_excel(path)
@@ -1247,6 +1253,7 @@ def test_from_excel_sheetname_title():
     "sheet_name",
     ["Sheet1", "AnotherSpecialName", "SpecialName", "SecondSpecialName", 0, 1, 2, 3],
 )
+@check_file_leaks
 def test_from_excel_sheet_name(sheet_name):
     fname = "modin/pandas/test/data/modin_error_book.xlsx"
     modin_df = pd.read_excel(fname, sheet_name=sheet_name)
diff --git a/modin/pandas/test/utils.py b/modin/pandas/test/utils.py
--- a/modin/pandas/test/utils.py
+++ b/modin/pandas/test/utils.py
@@ -23,11 +23,13 @@
 )
 import modin.pandas as pd
 from modin.utils import to_pandas
-from modin.config import TestDatasetSize
+from modin.config import TestDatasetSize, TrackFileLeaks
 from io import BytesIO
 import os
 from string import ascii_letters
 import csv
+import psutil
+import functools
 
 random_state = np.random.RandomState(seed=42)
 
@@ -1001,3 +1003,38 @@ def insert_lines_to_csv(
             **csv_reader_writer_params,
         )
         writer.writerows(lines)
+
+
+def _get_open_files():
+    """
+    psutil open_files() can return a lot of extra information that we can allow to
+    be different, like file position; for simplicity we care about path and fd only.
+    """
+    return sorted((info.path, info.fd) for info in psutil.Process().open_files())
+
+
+def check_file_leaks(func):
+    """
+    A decorator that ensures that no *newly* opened file handles are left
+    after decorated function is finished.
+    """
+    if not TrackFileLeaks.get():
+        return func
+
+    @functools.wraps(func)
+    def check(*a, **kw):
+        fstart = _get_open_files()
+        try:
+            return func(*a, **kw)
+        finally:
+            leaks = []
+            for item in _get_open_files():
+                try:
+                    fstart.remove(item)
+                except ValueError:
+                    leaks.append(item)
+            assert (
+                not leaks
+            ), f"Unexpected open handles left for: {', '.join(item[0] for item in leaks)}"
+
+    return check