CI: Add print skipped tests into azure step
State: closed

- [x] closes #26695 
- [ ] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=h1) Report
> Merging [#26698](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/cf25c5cd3ab1f647f780167df622c74b737fa8f5?src=pr&el=desc) will **decrease** coverage by `50.58%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26698/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #26698       +/-   ##
===========================================
- Coverage   91.78%    41.2%   -50.59%     
===========================================
  Files         174      179        +5     
  Lines       50703    50767       +64     
===========================================
- Hits        46538    20918    -25620     
- Misses       4165    29849    +25684
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `41.2% <ø> (-0.7%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.37%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.16%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.1%)` | :arrow_down: |
| [pandas/core/sparse/scipy\_sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL3NjaXB5X3NwYXJzZS5weQ==) | `10.14% <0%> (-89.86%)` | :arrow_down: |
| [pandas/core/tools/numeric.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvbnVtZXJpYy5weQ==) | `10.14% <0%> (-89.86%)` | :arrow_down: |
| ... and [139 more](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=footer). Last update [cf25c5c...68c39a1](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=h1) Report
> Merging [#26698](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/649ad5c91aabb95d7f200eec84b82cae1459fc65?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26698/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26698      +/-   ##
==========================================
- Coverage   91.88%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50701    50701              
==========================================
- Hits        46588    46583       -5     
- Misses       4113     4118       +5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.41% <ø> (ø)` | :arrow_up: |
| #single | `41.91% <ø> (-0.11%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26698/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.6% <0%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=footer). Last update [649ad5c...e06ebb8](https://codecov.io/gh/pandas-dev/pandas/pull/26698?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

This is the checks job, only the `validate_docstrings.py` tests are run here, and no tests are skipped there.

This is trickier than what it looks I'd say, ideally, I think we should unify how the tests are called, and add the printing of the skipped tests there. So the output is consistent among builds.

Closing this PR, but feel free to open a new one with those changes.
CC: @jorisvandenbossche 
@datapythonista It's not that this "simply" needs to be moved to a step in the posix.yml template? The tests are there always called with `ci/run_tests.sh`, so not that what should be unified.
I think the best approach would be to have this in `ci/run_tests.sh`. The print skipped is already in the travis docs, and in azure we'd probably like to add it to posix but also windows.

I'm ok with moving the code in this PR to `posix.yml` for now, but ideally I think we should avoid repeating it in every CI system.
Isn't it actually valuable that it is a separate step? That makes it simpler to look at it in the Azure interface (one of its advantages that you can split the log)
It would maybe be nice to move the `if` checks (for single multiple tests) to a common script, but I don't see the problem with repeating calling out to that script in each CI. We also repeat the call to `ci/run_tests.sh`.

Anyway, I think this provides enough short term value (eg for checking if the clipboard tests ran or not) to add like this for now.
Good point, then I'd simply have the `if` statement for the single/multi inside the script, and not in the CI config.

@xcz011 do you mind doing that?

The changes would be:
- Check for which of the files exist (the `if` in your code) inside `ci/print_skipped.py`, so we can simply call the script in the CI config
- Update `.travis.yml` to remove the `if`
- Add your changes (without the `if`) to `ci/azure/posix.yml` and `ci/azure/windows.yml`

Reopening this, so you can keep working on the same branch if you want.

Thanks!
> Good point, then I'd simply have the `if` statement for the single/multi inside the script, and not in the CI config.
> 
> @xcz011 do you mind doing that?
> 
> The changes would be:
> 
> * Check for which of the files exist (the `if` in your code) inside `ci/print_skipped.py`, so we can simply call the script in the CI config
> * Update `.travis.yml` to remove the `if`
> * Add your changes (without the `if`) to `ci/azure/posix.yml` and `ci/azure/windows.yml`
> 
> Reopening this, so you can keep working on the same branch if you want.
> 
> Thanks!



> Good point, then I'd simply have the `if` statement for the single/multi inside the script, and not in the CI config.
> 
> @xcz011 do you mind doing that?
> 
> The changes would be:
> 
> * Check for which of the files exist (the `if` in your code) inside `ci/print_skipped.py`, so we can simply call the script in the CI config
> * Update `.travis.yml` to remove the `if`
> * Add your changes (without the `if`) to `ci/azure/posix.yml` and `ci/azure/windows.yml`
> 
> Reopening this, so you can keep working on the same branch if you want.
> 
> Thanks!

Sure, I will try to do it and may need some help 
@xcz011 thanks a lot!
> @xcz011 thanks a lot!

Thanks all the helps !