CI/TST: Making ci/run_tests.sh fail if one of the steps fail
State: closed

Looks like when simplifying the running of the tests in the CI (#23924), I missed the `-e` in the bash header. And that makes the `ci/run_tests.sh` exit with status code 0, even if the calls to pytests fail.

This left the CI in green, even when tests fail for the last 3 days (sorry about that). I think nothing is broken.

This PR fixes the problem.

CC: @pandas-dev/pandas-core 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
Seems reasonable to me.  Wondering though if we should double check by adding a failing test just to test that this fails as expected?
I tested locally. I'd personally merge this asap, as the test of all PRs are failing silently (unless you check the logs). And we'll see quickly if this works as expected as soon as other PR tests fail.

But I surely can break something here and test if you think it's worth the delay.
> But I surely can break something here and test if you think it's worth the delay.

Fair enough.  I think we all can be vigilant to notice if it doesn't work. 🙂 
Any idea what the failure is from @datapythonista? https://travis-ci.org/pandas-dev/pandas/jobs/462954286#L1845

```
$ ci/run_tests.sh
============================= test session starts ==============================
platform linux2 -- Python 2.7.15, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /home/travis/build/pandas-dev/pandas, inifile: setup.cfg
plugins: xdist-1.24.1, forked-0.2, hypothesis-3.68.0
gw0 [0]
scheduling tests via LoadScheduling
 generated xml file: /home/travis/build/pandas-dev/pandas/test-data-single.xml -
======================== no tests ran in 84.95 seconds =========================
The command "ci/run_tests.sh" exited with 5.
```

Both of those failures are the slow / single tests, right?
Looks like when no tests are found with a `-m` pattern (the case for single and slow), the pytest process exits with status code 5. I'm checking what's the best option to fix it.
I see. Do we ever want to run `single and slow` though? I would have thought that it'd be `single or slow`.

edit: no that's not quite right... sorry.
So to run all the SLOW jobs, we need

`-n 1 -m slow and single`

and then

`-n 2 -m slow and not single`


And the problem is that `-m slow and single` is empty? As a workaround, can we add a dummy test with both those marks?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=h1) Report
> Merging [#24075](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/08395af43b813b69b1cf038d260104cfeeb3e787?src=pr&el=desc) will **increase** coverage by `49.86%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24075/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #24075       +/-   ##
===========================================
+ Coverage   42.38%   92.25%   +49.86%     
===========================================
  Files         161      161               
  Lines       51701    51701               
===========================================
+ Hits        21914    47696    +25782     
+ Misses      29787     4005    -25782
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.65% <ø> (?)` | |
| #single | `42.38% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/computation/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vcHl0YWJsZXMucHk=) | `92.37% <0%> (+0.3%)` | :arrow_up: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `92.3% <0%> (+0.92%)` | :arrow_up: |
| [pandas/util/\_test\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX3Rlc3RfZGVjb3JhdG9ycy5weQ==) | `93.24% <0%> (+4.05%)` | :arrow_up: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `58.36% <0%> (+8.17%)` | :arrow_up: |
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `99.24% <0%> (+9.84%)` | :arrow_up: |
| [pandas/core/reshape/util.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS91dGlsLnB5) | `100% <0%> (+11.53%)` | :arrow_up: |
| [pandas/compat/numpy/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9fX2luaXRfXy5weQ==) | `92.85% <0%> (+14.28%)` | :arrow_up: |
| [pandas/core/computation/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vY29tbW9uLnB5) | `85.71% <0%> (+14.28%)` | :arrow_up: |
| [pandas/core/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXBpLnB5) | `100% <0%> (+14.81%)` | :arrow_up: |
| [pandas/core/indexes/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9hcGkucHk=) | `99% <0%> (+14.85%)` | :arrow_up: |
| ... and [119 more](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=footer). Last update [08395af...0a67310](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=h1) Report
> Merging [#24075](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/08395af43b813b69b1cf038d260104cfeeb3e787?src=pr&el=desc) will **increase** coverage by `49.7%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24075/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24075      +/-   ##
==========================================
+ Coverage   42.38%   92.09%   +49.7%     
==========================================
  Files         161      161              
  Lines       51701    52102     +401     
==========================================
+ Hits        21914    47983   +26069     
+ Misses      29787     4119   -25668
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.66% <ø> (?)` | |
| #single | `42.81% <ø> (+0.43%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/computation/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vcHl0YWJsZXMucHk=) | `92.37% <0%> (+0.3%)` | :arrow_up: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `92.3% <0%> (+0.92%)` | :arrow_up: |
| [pandas/util/\_test\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX3Rlc3RfZGVjb3JhdG9ycy5weQ==) | `93.24% <0%> (+4.05%)` | :arrow_up: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `58.36% <0%> (+8.17%)` | :arrow_up: |
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `99.24% <0%> (+9.84%)` | :arrow_up: |
| [pandas/core/reshape/util.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS91dGlsLnB5) | `100% <0%> (+11.53%)` | :arrow_up: |
| [pandas/core/dtypes/dtypes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2R0eXBlcy5weQ==) | `89.82% <0%> (+13.66%)` | :arrow_up: |
| [pandas/compat/numpy/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9fX2luaXRfXy5weQ==) | `92.85% <0%> (+14.28%)` | :arrow_up: |
| [pandas/core/computation/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vY29tbW9uLnB5) | `85.71% <0%> (+14.28%)` | :arrow_up: |
| [pandas/core/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXBpLnB5) | `100% <0%> (+14.81%)` | :arrow_up: |
| ... and [119 more](https://codecov.io/gh/pandas-dev/pandas/pull/24075/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=footer). Last update [08395af...07375c7](https://codecov.io/gh/pandas-dev/pandas/pull/24075?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

That's correct @TomAugspurger, looks like it's a common pytest issue. The pytest devs propose the workaround I pushed. Not the nicest code, but this should work for now, and I'll try to find a cleaner approach for that script later (I don't like calling pytest twice, will try to avoid that).
LGTM.

On Mon, Dec 3, 2018 at 3:44 PM Marc Garcia <notifications@github.com> wrote:

> That's correct @TomAugspurger <https://github.com/TomAugspurger>, looks
> like it's a common pytest issue. The pytest devs propose the workaround I
> pushed. Not the nicest code, but this should work for now, and I'll try to
> find a cleaner approach for that script later (I don't like calling pytest
> twice, will try to avoid that).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24075#issuecomment-443881887>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIh6DxgCCo7bAnCFs4m-IAOKvVRJMks5u1ZtSgaJpZM4Y_L32>
> .
>

Can't really understand the errors in travis. They are surely unrelated, they don't seem to happen in master, and I can't reproduce them locally. @TomAugspurger have you seen this error before?
```
_____________________ TestTSPlot.test_is_error_nozeroindex _____________________
[gw0] linux -- Python 3.6.6 /home/travis/miniconda3/envs/pandas-dev/bin/python
self = <pandas.tests.plotting.test_datetimelike.TestTSPlot object at 0x7fca6c1875c0>
    def test_is_error_nozeroindex(self):
        # GH11858
        i = np.array([1, 2, 3])
        a = DataFrame(i, index=i)
>       _check_plot_works(a.plot, xerr=a)
pandas/tests/plotting/test_datetimelike.py:89: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/tests/plotting/test_datetimelike.py:1562: in _check_plot_works
    pickle.dump(fig, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
m = <bound method AffineBase.transform of <matplotlib.transforms.BlendedAffine2D object at 0x7fca6c567240>>
    def _pickle_method(m):
        """Handle pickling issues with class instance."""
>       if m.im_self is None:
E       AttributeError: 'function' object has no attribute 'im_self'
../../../miniconda3/envs/pandas-dev/lib/python3.6/site-packages/statsmodels/graphics/functional.py:32: AttributeError
```
I've seen them locally sometimes, but never figured out why. Seems like a weird matplotlib dependency on *something*.
That's weird, not sure why matplotlib or statmodels should fail randomly.

In any case, all green now. I'll let someone else review it before merging, as I made some changes since last reviews.