STYLE: Fixing and refactoring linting
State: closed

- Wrong path `pandas/src` corrected (`.px.in` was not being tested)
- Made more compact and readable
- Unified the flake8 errors that are being ignored
- Moving doctests and check_imports to `lint.sh`
- Moving linting from py2.7 to py3.6

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
There is something weird in travis. When I run `./ci/lint.sh` locally, I've got errors for all the `E722 do not use bare except` in python files. But those seem to be ignored when the script runs in Travis. When I tested the linting in azure, I've also got the errors.

@jreback @TomAugspurger any idea why is this happening? I couldn't see anything on the travis configuration or in `setup.cfg` that explains this.

Also, which is the desired behavior, do we want to allow base except?

<details>
inside ./ci/lint.sh
Linting .py code
pandas\compat\pickle_compat.py:36:13: E722 do not use bare except'
pandas\compat\pickle_compat.py:47:13: E722 do not use bare except'
pandas\compat\pickle_compat.py:185:1: E722 do not use bare except'
pandas\compat\pickle_compat.py:213:5: E722 do not use bare except'
pandas\core\frame.py:3263:13: E722 do not use bare except'
pandas\core\frame.py:7706:9: E722 do not use bare except'
pandas\core\indexing.py:2149:9: E722 do not use bare except'
pandas\core\indexing.py:2708:13: E722 do not use bare except'
pandas\core\indexing.py:2714:9: E722 do not use bare except'
pandas\core\nanops.py:506:13: E722 do not use bare except'
pandas\core\nanops.py:818:13: E722 do not use bare except'
pandas\core\ops.py:1548:17: E722 do not use bare except'
pandas\core\window.py:2507:5: E722 do not use bare except'
pandas\core\computation\pytables.py:414:9: E722 do not use bare except'
pandas\core\dtypes\common.py:470:5: E722 do not use bare except'
pandas\core\dtypes\dtypes.py:361:9: E722 do not use bare except'
pandas\core\dtypes\dtypes.py:522:13: E722 do not use bare except'
pandas\core\indexes\frozen.py:142:9: E722 do not use bare except'
pandas\core\indexes\multi.py:1005:17: E722 do not use bare except'
pandas\core\indexes\multi.py:1010:17: E722 do not use bare except'
pandas\core\indexes\multi.py:1689:9: E722 do not use bare except'
pandas\core\indexes\multi.py:2318:17: E722 do not use bare except'
pandas\core\indexes\multi.py:2821:17: E722 do not use bare except'
pandas\core\internals\blocks.py:669:9: E722 do not use bare except'
pandas\core\internals\blocks.py:1145:9: E722 do not use bare except'
pandas\core\internals\blocks.py:1160:9: E722 do not use bare except'
pandas\core\internals\blocks.py:2441:13: E722 do not use bare except'
pandas\core\internals\blocks.py:3175:9: E722 do not use bare except'
pandas\core\sparse\array.py:309:9: E722 do not use bare except'
pandas\core\tools\datetimes.py:244:17: E722 do not use bare except'
pandas\core\tools\datetimes.py:334:9: E722 do not use bare except'
pandas\core\tools\datetimes.py:731:5: E722 do not use bare except'
pandas\core\tools\datetimes.py:738:5: E722 do not use bare except'
pandas\core\tools\datetimes.py:745:5: E722 do not use bare except'
pandas\io\clipboards.py:45:9: E722 do not use bare except'
pandas\io\packers.py:706:13: E722 do not use bare except'
pandas\io\parsers.py:1811:9: E722 do not use bare except'
pandas\io\parsers.py:3037:13: E722 do not use bare except'
pandas\io\parsers.py:3266:9: E722 do not use bare except'
pandas\io\parsers.py:3287:9: E722 do not use bare except'
pandas\io\parsers.py:3291:9: E722 do not use bare except'
pandas\io\pickle.py:172:13: E722 do not use bare except'
pandas\io\pickle.py:177:5: E722 do not use bare except'
pandas\io\pytables.py:261:9: E722 do not use bare except'
pandas\io\pytables.py:398:5: E722 do not use bare except'
pandas\io\pytables.py:402:9: E722 do not use bare except'
pandas\io\pytables.py:520:9: E722 do not use bare except'
pandas\io\pytables.py:678:17: E722 do not use bare except'
pandas\io\pytables.py:1164:9: E722 do not use bare except'
pandas\io\pytables.py:1273:9: E722 do not use bare except'
pandas\io\pytables.py:1310:17: E722 do not use bare except'
pandas\io\pytables.py:1321:13: E722 do not use bare except'
pandas\io\pytables.py:1357:17: E722 do not use bare except'
pandas\io\pytables.py:1362:9: E722 do not use bare except'
pandas\io\pytables.py:1627:9: E722 do not use bare except'
pandas\io\pytables.py:1659:9: E722 do not use bare except'
pandas\io\pytables.py:1872:9: E722 do not use bare except'
pandas\io\pytables.py:2235:17: E722 do not use bare except'
pandas\io\pytables.py:2328:9: E722 do not use bare except'
pandas\io\pytables.py:2772:17: E722 do not use bare except'
pandas\io\pytables.py:2846:9: E722 do not use bare except'
pandas\io\pytables.py:2964:9: E722 do not use bare except'
pandas\io\pytables.py:3498:13: E722 do not use bare except'
pandas\io\pytables.py:3617:17: E722 do not use bare except'
pandas\io\pytables.py:3645:17: E722 do not use bare except'
pandas\io\pytables.py:4463:5: E722 do not use bare except'
pandas\io\pytables.py:4785:13: E722 do not use bare except'
pandas\io\sql.py:385:5: E722 do not use bare except'
pandas\io\sql.py:850:13: E722 do not use bare except'
pandas\io\sql.py:1363:9: E722 do not use bare except'
pandas\io\stata.py:1255:9: E722 do not use bare except'
pandas\io\stata.py:1260:9: E722 do not use bare except'
pandas\io\formats\console.py:103:5: E722 do not use bare except'
pandas\io\formats\console.py:121:5: E722 do not use bare except'
pandas\io\formats\console.py:140:5: E722 do not use bare except'
pandas\io\formats\console.py:152:5: E722 do not use bare except'
pandas\io\formats\terminal.py:81:5: E722 do not use bare except'
pandas\io\formats\terminal.py:111:5: E722 do not use bare except'
pandas\io\formats\terminal.py:123:9: E722 do not use bare except'
pandas\io\formats\terminal.py:132:9: E722 do not use bare except'
pandas\io\formats\terminal.py:138:9: E722 do not use bare except'
pandas\io\sas\sasreader.py:49:9: E722 do not use bare except'
pandas\io\sas\sas_xport.py:249:13: E722 do not use bare except'
pandas\tests\test_multilevel.py:1378:9: E722 do not use bare except'
pandas\tests\test_nanops.py:144:9: E722 do not use bare except'
pandas\tests\test_nanops.py:149:9: E722 do not use bare except'
pandas\tests\test_nanops.py:170:21: E722 do not use bare except'
pandas\tests\test_nanops.py:174:21: E722 do not use bare except'
pandas\tests\test_panel.py:338:13: E722 do not use bare except'
pandas\tests\test_panel.py:344:13: E722 do not use bare except'
pandas\tests\test_strings.py:2663:13: E722 do not use bare except'
pandas\tests\indexing\common.py:154:13: E722 do not use bare except'
pandas\tests\indexing\common.py:217:17: E722 do not use bare except'
pandas\tests\io\test_pytables.py:54:9: E722 do not use bare except'
pandas\tests\io\test_pytables.py:62:5: E722 do not use bare except'
pandas\tests\io\test_pytables.py:120:5: E722 do not use bare except'
pandas\tests\io\test_pytables.py:4624:21: E722 do not use bare except'
pandas\tests\io\test_sql.py:1793:9: E722 do not use bare except'
pandas\tests\io\test_sql.py:2378:9: E722 do not use bare except'
pandas\tests\io\test_sql.py:2405:9: E722 do not use bare except'
pandas\tests\io\formats\test_format.py:73:5: E722 do not use bare except'
pandas\tests\io\formats\test_format.py:455:13: E722 do not use bare except'
pandas\tseries\holiday.py:297:5: E722 do not use bare except'
pandas\tseries\holiday.py:429:9: E722 do not use bare except'
pandas\tseries\holiday.py:438:9: E722 do not use bare except'
pandas\util\_print_versions.py:24:9: E722 do not use bare except'
pandas\util\_print_versions.py:53:5: E722 do not use bare except'
pandas\util\_print_versions.py:111:9: E722 do not use bare except'
pandas\util\_validators.py:62:9: E722 do not use bare except'
Linting .py code DONE
Linting .pyx code
Linting .pyx code DONE
Linting .pxd and .pxi.in
Linting .pxd and .pxi.in DONE
Linting setup.py
Linting setup.py DONE
Linting scripts
Linting scripts DONE
Linting asv benchmarks
Linting asv benchmarks DONE
Linting doc scripts
doc/source/conf.py:568:9: E722 do not use bare except'
doc/source/conf.py:573:5: E722 do not use bare except'
doc/source/conf.py:580:5: E722 do not use bare except'
Linting doc scripts DONE
Linting .c and .h
Linting .c and .h DONE
Check for non-standard imports
Check for non-standard imports DONE
Check for pytest warns
Check for pytest warns DONE
Check for invalid testing
Check for invalid testing DONE
Check for invalid EA testing
Check for invalid EA testing DONE
Check for deprecated messages without sphinx directive
Check for deprecated messages without sphinx directive DONE
Check for old-style classes
Check for old-style classes DONE
Check for backticks incorrectly rendering because of missing spaces
Check for backticks incorrectly rendering because of missing spaces DONE
Check for incorrect sphinx directives
Check for incorrect sphinx directives DONE
</details>
we fix the pytest version currently 
there is an issue about this 3.4) i think

fixing these is separate and out of scope for this issue
This is the issue: #18419, thanks for pointing out.

Agree that's for a different issue.
The flake8 errors that we were ignoring was a bit inconsistent. I changed it, so we always ignore the same (the ones in `setup.cfg`), and I removed the ones that weren't required. I explicitly add E722 (bare except) to the list of ignored ones. I guess eventually we can start validating it, and some of the others.

This PR should be ready to be merged from my side.
Hello @datapythonista! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`ci/print_versions.py`](https://github.com/datapythonista/pandas/blob/adcda478c3af1ae0edeaf09630ce2c3d6926b36a/ci/print_versions.py) !


# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=h1) Report
> Merging [#22863](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d115900f4f80d2f9016b41041bde1564980415b3?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22863/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22863      +/-   ##
==========================================
- Coverage   92.19%   92.18%   -0.01%     
==========================================
  Files         169      169              
  Lines       50827    50830       +3     
==========================================
  Hits        46860    46860              
- Misses       3967     3970       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.6% <ø> (-0.01%)` | :arrow_down: |
| #single | `42.37% <ø> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/plotting/\_core.py](https://codecov.io/gh/pandas-dev/pandas/pull/22863/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb3JlLnB5) | `83.35% <0%> (-0.2%)` | :arrow_down: |
| [pandas/plotting/\_compat.py](https://codecov.io/gh/pandas-dev/pandas/pull/22863/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb21wYXQucHk=) | `91.3% <0%> (+0.39%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=footer). Last update [d115900...adcda47](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=h1) Report
> Merging [#22863](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/a4482db46535b018053b4351689ccc3c79257d8c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22863/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #22863   +/-   ##
=======================================
  Coverage   92.19%   92.19%           
=======================================
  Files         169      169           
  Lines       50904    50904           
=======================================
  Hits        46933    46933           
  Misses       3971     3971
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.61% <ø> (ø)` | :arrow_up: |
| #single | `42.31% <ø> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/22863/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `93.87% <0%> (-0.01%)` | :arrow_down: |
| [pandas/io/packers.py](https://codecov.io/gh/pandas-dev/pandas/pull/22863/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhY2tlcnMucHk=) | `88.04% <0%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/22863/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.11% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/range.py](https://codecov.io/gh/pandas-dev/pandas/pull/22863/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9yYW5nZS5weQ==) | `95.73% <0%> (ø)` | :arrow_up: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/22863/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `92.44% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=footer). Last update [a4482db...8738e80](https://codecov.io/gh/pandas-dev/pandas/pull/22863?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jorisvandenbossche I moved the doctests to the `lint.sh` script in this case. Hopefully soon we'll have all them passing, and we should be able to run all them with a single command, and not file by file. Also, some of the code in `lint.sh` is the same as in `doctests.sh`, so it made sense to me.

Can you let me know if this looks ok.

@pandas-dev/pandas-core after this is merged, my idea is to move the linting to azure, as the tests in #22854 IMO show a much better interface to identify the errors than the travis logs: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=367&view=logs

And I'm still researching how to do it, but we should be able to show the errors without entering the logs (where now says `Bash exited with code '1'.`, and also link to the line in the code where the linting problem happened).

Let me know if there is any reason I'm missing to not make the change.  
@jreback I finally added a parameter to the script (I think there are already too many files in `ci/` to add 3 more), so every section can be called independently. I also renamed it from `ci/lint.sh` to `ci/code_checks.sh` as (technically speaking) linting is just one of the things that it does. So:
- `./ci/code_checks.sh` <- runs everything
- `./ci/code_checks.sh lint` <- flake8 and cpplint
- `./ci/code_checks.sh patterns` <- greps
- `./ci/code_checks.sh doctests` <- doctests

Let me know if this makes sense, or if there is anything else before this can be merged. Thanks
@jreback we have several PRs coming that fix doctests, and change `ci/doctests.sh` which has been removed here. Can you take a look and see if you're happy with the approach of using parameters for each part of the linting instead of splitting the file (it's described above)? I need to keep resolving conflicts if we don't merge this.
@datapythonista lgtm to me now. just failing the checks:https://travis-ci.org/pandas-dev/pandas/jobs/439177405

can merge on green.