TST: Fix flaky import test
State: closed

I'm not sure why, but the missing dependency test is
causing issues. Now we check that things work by running
it in a subprocess with site-packages disabled.

Closes https://github.com/pandas-dev/pandas/issues/26952

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=h1) Report
> Merging [#26953](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/baeb1bf92b763861212d0dee951bd8ea658deadb?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26953/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26953      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50746    50746              
==========================================
- Hits        46624    46619       -5     
- Misses       4122     4127       +5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <ø> (ø)` | :arrow_up: |
| #single | `41.1% <ø> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26953/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26953/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.89% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26953/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.84% <0%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=footer). Last update [baeb1bf...df1030b](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=h1) Report
> Merging [#26953](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/baeb1bf92b763861212d0dee951bd8ea658deadb?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26953/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26953      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50746    50746              
==========================================
- Hits        46624    46619       -5     
- Misses       4122     4127       +5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <ø> (ø)` | :arrow_up: |
| #single | `41.1% <ø> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26953/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26953/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.89% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26953/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.84% <0%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=footer). Last update [baeb1bf...df1030b](https://codecov.io/gh/pandas-dev/pandas/pull/26953?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I think the change here makes sense. But I'm missing something, the test I see failing is related but not the one fixed here:
```
=================================== FAILURES ===================================
______________________________ TestPDApi.test_api ______________________________
[gw3] darwin -- Python 3.5.6 /Users/vsts/miniconda3/envs/pandas-dev/bin/python

self = <pandas.tests.api.test_api.TestPDApi object at 0x12d4ec908>

    def test_api(self):
    
        self.check(pd,
                   self.lib + self.misc +
                   self.modules + self.deprecated_modules +
                   self.classes + self.deprecated_classes +
                   self.deprecated_classes_in_future +
                   self.funcs + self.funcs_option +
                   self.funcs_read + self.funcs_to +
                   self.deprecated_funcs_in_future +
                   self.deprecated_funcs + self.private_modules,
>                  self.ignored)

pandas/tests/api/test_api.py:113: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/tests/api/test_api.py:18: in check
    tm.assert_almost_equal(result, expected)
pandas/util/testing.py:325: in assert_almost_equal
    **kwargs)
pandas/_libs/testing.pyx:65: in pandas._libs.testing.assert_almost_equal
    cpdef assert_almost_equal(a, b,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise_assert_detail(obj, '{0} length are different'.format(obj),
E   AssertionError: Iterable are different
E   
E   Iterable length are different
E   [left]:  132
E   [right]: 129
E   [diff]: ['dependency', 'missing_dependencies', 'hard_dependencies']
```

I can't understand why it's failing, if I understand it correctly, the test says that `pandas` got the attributes `dependency`,... and it shouldn't. I see that those shouldn't exist, as they are removed not long after they're created: https://github.com/pandas-dev/pandas/blob/master/pandas/__init__.py#L17

Not sure how is that possible, the `import pandas` shouldn't finish until `__init__.py` is fully executed I'd say. And the import seems just a regular import at the top of the file: https://github.com/pandas-dev/pandas/blob/master/pandas/tests/api/test_api.py#L104

What it comes to my mind is... Since we don't want those attributes to be public attributes of `pandas`, wouldn't make sense to prefix them with an underscore? Regardless of the test I think that would be correct (and should fix the test too).

@jreback @jorisvandenbossche I guess you want to have a look here, this is affecting few PRs.
My guess is that there's a strange interaction between the test / changes in ea06f8d1157601b5fdb48598e27b02149828fba0 and the existing `test_api` tests. See

```python
import builtins
import pandas
import importlib


def mock_import_fail(name, *args, **kwargs):
    if name == "numpy":
        raise ImportError("cannot import name numpy")
    elif name == "pytz":
        raise ImportError("cannot import name some_dependency")
    elif name == "dateutil":
        raise ImportError("cannot import name some_other_dependency")
    else:
        return original_import(name, *args, **kwargs)


print(len(dir(pandas)))
builtins.__import__ = mock_import_fail
try:
    importlib.reload(pandas)
except:
    pass
print(len(dir(pandas)))
```

which outputs

```
141
144
```

So my guess is that if the missing required dep is run before the test_api, we fail.
Ah, here's a simpler reproducer

```
$ pytest pandas/tests/test_downstream.py::test_missing_required_dependency pandas/tests/api/test_api.py::TestPDApi::test_api

=================================================== FAILURES ====================================================
______________________________________________ TestPDApi.test_api _______________________________________________

self = <pandas.tests.api.test_api.TestPDApi object at 0x11f8ddcf8>

    def test_api(self):

        self.check(pd,
                   self.lib + self.misc +
                   self.modules + self.deprecated_modules +
                   self.classes + self.deprecated_classes +
                   self.deprecated_classes_in_future +
                   self.funcs + self.funcs_option +
                   self.funcs_read + self.funcs_to +
                   self.deprecated_funcs_in_future +
                   self.deprecated_funcs + self.private_modules,
>                  self.ignored)

pandas/tests/api/test_api.py:113:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas/tests/api/test_api.py:18: in check
    tm.assert_almost_equal(result, expected)
pandas/util/testing.py:325: in assert_almost_equal
    **kwargs)
pandas/_libs/testing.pyx:65: in pandas._libs.testing.assert_almost_equal
    cpdef assert_almost_equal(a, b,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   raise_assert_detail(obj, '{0} length are different'.format(obj),
E   AssertionError: Iterable are different
E
E   Iterable length are different
E   [left]:  132
E   [right]: 129
E   [diff]: ['dependency', 'missing_dependencies', 'hard_dependencies']

pandas/_libs/testing.pyx:163: AssertionError
```

I thought I tried that earlier, but apparently I did the wrong one. On my branch that passes.
I'm going to merge this since this is sometimes affecting other PRs. I'll be around all day to do fixups if needed.