TST: placement of network error catching in s3 tests
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19645?src=pr&el=h1) Report
> Merging [#19645](https://codecov.io/gh/pandas-dev/pandas/pull/19645?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/569bc7a35c06ef8eb9a121ca983ba8951399a124?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `60.46%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/19645/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/19645?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #19645      +/-   ##
==========================================
- Coverage   91.58%   91.56%   -0.03%     
==========================================
  Files         150      150              
  Lines       48806    48829      +23     
==========================================
+ Hits        44701    44709       +8     
- Misses       4105     4120      +15
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.93% <60.46%> (-0.03%)` | :arrow_down: |
| #single | `41.71% <6.97%> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/19645?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/19645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `91.07% <100%> (ø)` | :arrow_up: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/19645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `90.27% <100%> (ø)` | :arrow_up: |
| [pandas/io/packers.py](https://codecov.io/gh/pandas-dev/pandas/pull/19645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhY2tlcnMucHk=) | `87.31% <16.66%> (-1.34%)` | :arrow_down: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/19645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.32% <33.33%> (-0.25%)` | :arrow_down: |
| [pandas/io/s3.py](https://codecov.io/gh/pandas-dev/pandas/pull/19645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3MzLnB5) | `72.72% <50%> (-13.64%)` | :arrow_down: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/19645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `68.77% <50%> (-0.3%)` | :arrow_down: |
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/19645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `92.23% <75%> (-0.37%)` | :arrow_down: |
| [pandas/io/parquet.py](https://codecov.io/gh/pandas-dev/pandas/pull/19645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnF1ZXQucHk=) | `71.79% <80%> (-0.28%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/19645/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `83.85% <0%> (+0.2%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19645?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19645?src=pr&el=footer). Last update [569bc7a...825f2a8](https://codecov.io/gh/pandas-dev/pandas/pull/19645?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Hello @jreback! Thanks for updating the PR.





























Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on February 12, 2018 at 12:20 Hours UTC
This I believe exposes some issues that exist with the s3 tests and mocking that have our CI occasionaly fail. I think there is some interaction going on. When run separately these all work.

 ``pytest  pandas/tests/io/parser/test_network.py pandas/tests/io/json/test_pandas.py pandas/tests/io/test_parquet.py -r xXs -k s3 -v --tb=short``

xref #19585 

was trying to remove the resource closing issues (IOW on s3 access we don't close s3).

any ideas here: @jorisvandenbossche @TomAugspurger @maximveksler @mdurant



this fails for me locally - i was expecting it to fail in the CI actually 

let me take a closer look
So these pass now all pass, but have these warnings. any ideas?
```
Users/jreback/miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/dataframe.py:7: FutureWarning: 'pandas.core' is private. Use 'pandas.Categorical'
  from pandas.core.categorical import Categorical, CategoricalDtype
collected 140 items                                                                                                                                                                                                                         

pandas/tests/io/json/test_pandas.py .                                                                                                                                                                                                 [  7%]
pandas/tests/io/test_parquet.py ..                                                                                                                                                                                                    [ 21%]
pandas/tests/io/parser/test_network.py ...........                                                                                                                                                                                    [100%]

============================================================================================================= warnings summary ==============================================================================================================
pandas/tests/io/parser/test_network.py::TestS3::()::test_parse_public_s3_bucket
  /Users/jreback/pandas/pandas/io/s3.py:38: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.20.10.2', 52618), raddr=('52.216.96.141', 443)>
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)
  /Users/jreback/pandas/pandas/io/s3.py:38: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.20.10.2', 52626), raddr=('52.216.96.141', 443)>
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)
  /Users/jreback/pandas/pandas/io/s3.py:38: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.20.10.2', 52629), raddr=('52.216.96.141', 443)>
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)
  /Users/jreback/pandas/pandas/io/s3.py:38: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.20.10.2', 52630), raddr=('52.216.96.141', 443)>
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)

pandas/tests/io/parser/test_network.py::TestS3::()::test_parse_public_s3n_bucket
  /Users/jreback/pandas/pandas/io/s3.py:38: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.20.10.2', 52632), raddr=('52.216.96.141', 443)>
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)

pandas/tests/io/parser/test_network.py::TestS3::()::test_parse_public_s3a_bucket
  /Users/jreback/pandas/pandas/io/s3.py:38: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.20.10.2', 52633), raddr=('52.216.96.141', 443)>
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)

pandas/tests/io/parser/test_network.py::TestS3::()::test_parse_public_s3_bucket_nrows
  /Users/jreback/pandas/pandas/io/s3.py:38: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.20.10.2', 52635), raddr=('52.216.96.141', 443)>
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)
  /Users/jreback/pandas/pandas/io/s3.py:38: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.20.10.2', 52640), raddr=('52.216.96.141', 443)>
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)
  /Users/jreback/pandas/pandas/io/s3.py:38: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.20.10.2', 52645), raddr=('52.216.96.141', 443)>
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)

pandas/tests/io/parser/test_network.py::TestS3::()::test_parse_public_s3_bucket_chunked
```
```diff
@@ -194,7 +197,8 @@ def get_filepath_or_buffer(filepath_or_buffer, encoding=None,
-        return reader, encoding, compression
+        return reader, encoding, compression, True
```
This resulted in a backward incompatible change for `pandas.io.common.get_filepath_or_buffer` between 0.22 and 0.23. I couldn't find anything about it in the release notes either. Unless it's a private function, but nothing in the import path suggests it...