Repository: pandas-dev/pandas

Issue:
BLD/IO: Elusive Travis ResourceWarning & closing files by default 
Every once in a while Travis will raise a `ResourceWarning`

```
____________________ TestPythonParser.test_no_header_prefix ____________________
[gw0] darwin -- Python 3.5.6 /Users/travis/miniconda3/envs/pandas/bin/python
self = <pandas.tests.io.parser.test_parsers.TestPythonParser object at 0x11d030da0>
    def test_no_header_prefix(self):
        data = """1,2,3,4,5
    6,7,8,9,10
    11,12,13,14,15
    """
        df_pref = self.read_table(StringIO(data), sep=',', prefix='Field'
>                                 header=None)
pandas/tests/io/parser/header.py:48:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas/tests/io/parser/test_parsers.py:111: in read_table
    df = read_table(*args, **kwds)
../../../miniconda3/envs/pandas/lib/python3.5/contextlib.py:66: in __exit__
    next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
E           AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedReader name='/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/tmpp8reunuatest_file.zip'>",), '/Users/travis/miniconda3/envs/pandas/lib/python3.5/site-packages/py/_vendored_packages/apipkg.py', 146)].
```

And currently our file closing mechanism for parser functions is like so:
https://github.com/pandas-dev/pandas/blob/73dd6ece3b66008eb8077098c101d0e300b1e714/pandas/io/parsers.py#L459-L463

It seems pointless having a generic `try`/`except` within an `if` statement (why have the if statement at all then?). More broadly, should we attempt to close all file-like objects anyways even if they are not necessarily needed (like StringIO in this Travis case)? Might help alleviate this Travis error.

Some examples:

- https://travis-ci.org/pandas-dev/pandas/jobs/427395352#L2258
- https://travis-ci.org/pandas-dev/pandas/jobs/427392469#L2259
CI: xlrd PendingDeprecationWarning
https://travis-ci.org/pandas-dev/pandas/jobs/427910477

```
E           AssertionError: Caused unexpected warning(s): [('PendingDeprecationWarning', PendingDeprecationWarning("This method will be removed in future versions.  Use 'tree.iter()' or 'list(tree.iter())' instead.",), '/home/travis/miniconda3/envs/pandas/lib/python2.7/site-packages/xlrd/xlsx.py', 246), ('PendingDeprecationWarning', PendingDeprecationWarning("This method will be removed in future versions.  Use 'tree.iter()' or 'list(tree.iter())' instead.",), '/home/travis/miniconda3/envs/pandas/lib/python2.7/site-packages/xlrd/xlsx.py', 292), ('PendingDeprecationWarning', PendingDeprecationWarning("This method will be removed in future versions.  Use 'tree.iter()' or 'list(tree.iter())' instead.",), '/home/travis/miniconda3/envs/pandas/lib/python2.7/site-packages/xlrd/xlsx.py', 246)].
```


Discussion:
I started to look into this yesterday, and will try again today. If anyone is able to reproduce locally, let me know.

The interesting part is that the errors always reference zip files like `tmpp8reunuatest_file.zip`, even though the failing test isn't a compression test. So an unrelated test is seemingly leaking a warning, and it's failing here?
Kept having lots of those as well, and commented about it in the closest-seeming issue I found (https://github.com/pandas-dev/pandas/issues/13962#issuecomment-405025631), including some links to failed builds (July 2018). 
> So an unrelated test is seemingly leaking a warning

Is it the warning that's coming from the unrelated test or just the unclosed file?
I was able to reproduce locally with `pytest -v -n 2 -m "not single" -r xX pandas/tests/io --count=5 `. So that's something....

> Is it the warning that's coming from the unrelated test or just the unclosed file?

It seems like the unclosed file warning has to be from a separate test right? In e.g. https://travis-ci.org/pandas-dev/pandas/jobs/427395352#L2258 we shouldn't be opening / closing a zip file.
> It seems like the unclosed file warning has to be from a separate test right? 

Yah, I'm just trying to rule out the extra-weird case where the ResourceWarning is being issued during the compression test but only showing up in the parser test, as opposed to the less-weird case where the compression test leaves a file open but that file only leads to a ResourceWarning later on.

Could check `psutil.open_files()` at the end of the compression test?
Gotcha, I suspect (hope) that the less-weird case is what's happening.


Resolving patch:
diff --git a/pandas/conftest.py b/pandas/conftest.py
--- a/pandas/conftest.py
+++ b/pandas/conftest.py
@@ -8,11 +8,14 @@
 import pandas as pd
 from pandas.compat import PY3
 import pandas.util._test_decorators as td
-
 import hypothesis
-hypothesis.settings.suppress_health_check = (hypothesis.HealthCheck.too_slow,)
-# HealthCheck.all() to disable all health checks
-# https://hypothesis.readthedocs.io/en/latest/healthchecks.html
+
+
+hypothesis.settings.register_profile(
+    "ci",
+    suppress_health_check=(hypothesis.HealthCheck.too_slow,)
+)
+hypothesis.settings.load_profile("ci")
 
 
 def pytest_addoption(parser):


Test patch:
diff --git a/pandas/tests/io/test_excel.py b/pandas/tests/io/test_excel.py
--- a/pandas/tests/io/test_excel.py
+++ b/pandas/tests/io/test_excel.py
@@ -105,6 +105,7 @@ def get_exceldf(self, basename, ext, *args, **kwds):
 class ReadingTestsBase(SharedItems):
     # This is based on ExcelWriterBase
 
+    @td.skip_if_no('xlrd', '1.0.1')  # GH-22682
     def test_usecols_int(self, ext):
 
         dfref = self.get_csv_refdf('test1')
@@ -122,6 +123,7 @@ def test_usecols_int(self, ext):
         tm.assert_frame_equal(df2, dfref, check_names=False)
         tm.assert_frame_equal(df3, dfref, check_names=False)
 
+    @td.skip_if_no('xlrd', '1.0.1')  # GH-22682
     def test_usecols_list(self, ext):
 
         dfref = self.get_csv_refdf('test1')
@@ -140,6 +142,7 @@ def test_usecols_list(self, ext):
         tm.assert_frame_equal(df2, dfref, check_names=False)
         tm.assert_frame_equal(df3, dfref, check_names=False)
 
+    @td.skip_if_no('xlrd', '1.0.1')  # GH-22682
     def test_usecols_str(self, ext):
 
         dfref = self.get_csv_refdf('test1')
@@ -219,6 +222,7 @@ def test_excel_passes_na(self, ext):
                              columns=['Test'])
         tm.assert_frame_equal(parsed, expected)
 
+    @td.skip_if_no('xlrd', '1.0.1')  # GH-22682
     def test_deprecated_sheetname(self, ext):
         # gh-17964
         excel = self.get_excelfile('test1', ext)
@@ -229,6 +233,7 @@ def test_deprecated_sheetname(self, ext):
         with pytest.raises(TypeError):
             read_excel(excel, sheet='Sheet1')
 
+    @td.skip_if_no('xlrd', '1.0.1')  # GH-22682
     def test_excel_table_sheet_by_index(self, ext):
 
         excel = self.get_excelfile('test1', ext)
@@ -507,6 +512,7 @@ def test_date_conversion_overflow(self, ext):
         result = self.get_exceldf('testdateoverflow', ext)
         tm.assert_frame_equal(result, expected)
 
+    @td.skip_if_no('xlrd', '1.0.1')  # GH-22682
     def test_sheet_name_and_sheetname(self, ext):
         # GH10559: Minor improvement: Change "sheet_name" to "sheetname"
         # GH10969: DOC: Consistent var names (sheetname vs sheet_name)
diff --git a/pandas/tests/io/test_pickle.py b/pandas/tests/io/test_pickle.py
--- a/pandas/tests/io/test_pickle.py
+++ b/pandas/tests/io/test_pickle.py
@@ -332,9 +332,9 @@ def compress_file(self, src_path, dest_path, compression):
             f = bz2.BZ2File(dest_path, "w")
         elif compression == 'zip':
             import zipfile
-            zip_file = zipfile.ZipFile(dest_path, "w",
-                                       compression=zipfile.ZIP_DEFLATED)
-            zip_file.write(src_path, os.path.basename(src_path))
+            f = zipfile.ZipFile(dest_path, "w",
+                                compression=zipfile.ZIP_DEFLATED)
+            f.write(src_path, os.path.basename(src_path))
         elif compression == 'xz':
             lzma = pandas.compat.import_lzma()
             f = lzma.LZMAFile(dest_path, "w")
@@ -345,7 +345,7 @@ def compress_file(self, src_path, dest_path, compression):
         if compression != "zip":
             with open(src_path, "rb") as fh:
                 f.write(fh.read())
-            f.close()
+        f.close()
 
     def test_write_explicit(self, compression, get_random_path):
         base = get_random_path
diff --git a/pandas/util/testing.py b/pandas/util/testing.py
--- a/pandas/util/testing.py
+++ b/pandas/util/testing.py
@@ -2560,7 +2560,7 @@ class for all warnings. To check that no warning is returned,
         the ``__warningsregistry__`` to ensure that no warning messages are
         suppressed by this context manager. If ``None`` is specified,
         the ``__warningsregistry__`` keeps track of which warnings have been
-         shown, and does not show them again.
+        shown, and does not show them again.
     check_stacklevel : bool, default True
         If True, displays the line that called the function containing
         the warning to show were the function is called. Otherwise, the
@@ -2589,7 +2589,7 @@ class for all warnings. To check that no warning is returned,
     with warnings.catch_warnings(record=True) as w:
 
         if clear is not None:
-            # make sure that we are clearning these warnings
+            # make sure that we are clearing these warnings
             # if they have happened before
             # to guarantee that we will catch them
             if not is_list_like(clear):