Drop Python 3.6 support
State: closed

- [x] closes #34472 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
The py37 minimum version CI times out after an hour. Not sure why

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=39023&view=logs&j=2d7fb38a-2053-50f3-a67c-09f6e91d3121&t=449937cc-3d50-56b5-5662-e489f41f1268
I bumped the minimum python version to 3.7.1. That fixed the CI timeout. I've tested it on my local machine. While using 3.7.0, running this test class `pandas/tests/io/json/test_json_table_schema.py::TestTableOrient` will cause a Segmentation fault.

```bash
(pandas-test) fangchenli@fangchenli-home-office:~/Workspace/pandas-VirosaLi$ pytest pandas/tests/io/json/test_json_table_schema.py::TestTableOrient
============================================================================================================================================= test session starts =============================================================================================================================================
platform linux -- Python 3.7.0, pytest-5.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/fangchenli/Workspace/pandas-VirosaLi, inifile: setup.cfg
plugins: forked-1.2.0, xdist-1.33.0, azurepipelines-0.8.0, hypothesis-5.19.1
collected 47 items                                                                                                                                                                                                                                                                                            

pandas/tests/io/json/test_json_table_schema.py ..........................................Fatal Python error: Segmentation fault

Current thread 0x00007f54ac64d740 (most recent call first):
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 166 in __getitem__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 646 in _parse
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 426 in _parse_sub
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 816 in _parse
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 426 in _parse_sub
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 930 in parse
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_compile.py", line 764 in compile
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/re.py", line 286 in _compile
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/re.py", line 234 in compile
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/dataclasses.py", line 196 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 728 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "/home/fangchenli/Workspace/pandas-VirosaLi/pandas/core/dtypes/inference.py", line 420 in is_dataclass
  File "/home/fangchenli/Workspace/pandas-VirosaLi/pandas/core/frame.py", line 500 in __init__
  File "/home/fangchenli/Workspace/pandas-VirosaLi/pandas/tests/io/json/test_json_table_schema.py", line 619 in test_timestamp_in_columns
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/python.py", line 165 in pytest_pyfunc_call
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/python.py", line 1451 in runtest
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 117 in pytest_runtest_call
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 192 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 220 in from_call
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 192 in call_runtest_hook
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 167 in call_and_report
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 87 in runtestprotocol
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 72 in pytest_runtest_protocol
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 278 in pytest_runtestloop
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 257 in _main
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 213 in wrap_session
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 250 in pytest_cmdline_main
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/config/__init__.py", line 74 in main
  File "/home/fangchenli/miniconda3/envs/pandas-test/bin/pytest", line 11 in <module>
Segmentation fault (core dumped)
```
I haven't pinpointed why exactly this happened. But it does disappear for Python 3.7.1.

Couldn't get the linux py38_np_dev ci to work. Need some help.
needs a release note in doc\source\whatsnew\v1.2.0.rst detailing changes in minimum versions. see 'Increased minimum versions for dependencies' in doc\source\whatsnew\v1.0.0.rst
> Couldn't get the linux py38_np_dev ci to work. Need some help.

these are maybe the known failures (should be 20 at this time, there are 29 here, merge upstream/master to make sure)
I set the minimal python version to 3.7.1 for now. See #35309. We could lower it to 3.7.0 when we fixed that.
merge master and see if can get the py38_np_dev build passing. ping on green. also have a couple of commented builds?
Hello @VirosaLi! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-10 22:21:11 UTC
Those two tests
 ```
pandas\tests\io\parser\test_common.py:test_chunks_have_consistent_numerical_type
pandas\tests\io\parser\test_common.py:test_warn_if_chunks_have_mismatched_type
```
produces `ResourceWarning` occasionally on Windows-py37-np16 and travis-37. I skipped them.

On the NumPy dev build. The following tests:
```bash
FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_to_json
FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_date_format_raises
FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_datetime_tz
FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts0]
FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts1]
FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts2]
FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range0]
FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range1]
FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range2]
```

produce `DeprecationWarning: an integer is required (got type float).  Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.`

I filtered this warning for now.

I'll open separate issues for those two warnings after this PR is merged.
> Those two tests
> 
> ```
> pandas\tests\io\parser\test_common.py:test_chunks_have_consistent_numerical_type
> pandas\tests\io\parser\test_common.py:test_warn_if_chunks_have_mismatched_type
> ```
> 
> produces `ResourceWarning` occasionally on Windows-py37-np16 and travis-37. I skipped them.
> 
> On the NumPy dev build. The following tests:
> 
> ```shell
> FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_to_json
> FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_date_format_raises
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_datetime_tz
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts0]
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts1]
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts2]
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range0]
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range1]
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range2]
> ```
> 
> produce `DeprecationWarning: an integer is required (got type float). Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.`
> 
> I filtered this warning for now.
> 
> I'll open separate issues for those two warnings after this PR is merged.

@VirosaLi thanks.

so looks all good. what is the status of the 32-bit build?
> > Those two tests
> > ```
> > pandas\tests\io\parser\test_common.py:test_chunks_have_consistent_numerical_type
> > pandas\tests\io\parser\test_common.py:test_warn_if_chunks_have_mismatched_type
> > ```
> > 
> > 
> > produces `ResourceWarning` occasionally on Windows-py37-np16 and travis-37. I skipped them.
> > On the NumPy dev build. The following tests:
> > ```shell
> > FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_to_json
> > FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_date_format_raises
> > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_datetime_tz
> > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts0]
> > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts1]
> > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts2]
> > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range0]
> > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range1]
> > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range2]
> > ```
> > 
> > 
> > produce `DeprecationWarning: an integer is required (got type float). Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.`
> > I filtered this warning for now.
> > I'll open separate issues for those two warnings after this PR is merged.
> 
> @VirosaLi thanks.
> 
> so looks all good. what is the status of the 32-bit build?

It failed. So I left it commented. We could open another issue for the 32-bit build. 
> > > Those two tests
> > > ```
> > > pandas\tests\io\parser\test_common.py:test_chunks_have_consistent_numerical_type
> > > pandas\tests\io\parser\test_common.py:test_warn_if_chunks_have_mismatched_type
> > > ```
> > > 
> > > 
> > > produces `ResourceWarning` occasionally on Windows-py37-np16 and travis-37. I skipped them.
> > > On the NumPy dev build. The following tests:
> > > ```shell
> > > FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_to_json
> > > FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_date_format_raises
> > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_datetime_tz
> > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts0]
> > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts1]
> > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts2]
> > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range0]
> > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range1]
> > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range2]
> > > ```
> > > 
> > > 
> > > produce `DeprecationWarning: an integer is required (got type float). Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.`
> > > I filtered this warning for now.
> > > I'll open separate issues for those two warnings after this PR is merged.
> > 
> > 
> > @VirosaLi thanks.
> > so looks all good. what is the status of the 32-bit build?
> 
> It failed. So I left it commented. We could open another issue for the 32-bit build.

do you recall how much work to get it running?

ok on opening an issue, can you followup on that build?
@TomAugspurger any objections to merging then doing the 32-bit build after?
> > > > Those two tests
> > > > ```
> > > > pandas\tests\io\parser\test_common.py:test_chunks_have_consistent_numerical_type
> > > > pandas\tests\io\parser\test_common.py:test_warn_if_chunks_have_mismatched_type
> > > > ```
> > > > 
> > > > 
> > > > produces `ResourceWarning` occasionally on Windows-py37-np16 and travis-37. I skipped them.
> > > > On the NumPy dev build. The following tests:
> > > > ```shell
> > > > FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_to_json
> > > > FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_date_format_raises
> > > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_datetime_tz
> > > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts0]
> > > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts1]
> > > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts2]
> > > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range0]
> > > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range1]
> > > > FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range2]
> > > > ```
> > > > 
> > > > 
> > > > produce `DeprecationWarning: an integer is required (got type float). Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.`
> > > > I filtered this warning for now.
> > > > I'll open separate issues for those two warnings after this PR is merged.
> > > 
> > > 
> > > @VirosaLi thanks.
> > > so looks all good. what is the status of the 32-bit build?
> > 
> > 
> > It failed. So I left it commented. We could open another issue for the 32-bit build.
> 
> do you recall how much work to get it running?
> 
> ok on opening an issue, can you followup on that build?

I'm not familiar with the 32-bit build. If I remember it correctly, it has been commented out for a while. I could work on it later this week.
I'm OK with dropping the 32-bit build. We need to restructure how it's done
anyway (since Anaconda isn't building 32-bit packages anymore).

On Mon, Aug 10, 2020 at 7:36 PM Fangchen Li <notifications@github.com>
wrote:

> Those two tests
>
> pandas\tests\io\parser\test_common.py:test_chunks_have_consistent_numerical_type
> pandas\tests\io\parser\test_common.py:test_warn_if_chunks_have_mismatched_type
>
> produces ResourceWarning occasionally on Windows-py37-np16 and travis-37.
> I skipped them.
> On the NumPy dev build. The following tests:
>
> FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_to_json
> FAILED pandas/tests/io/json/test_json_table_schema.py::TestTableOrient::test_date_format_raises
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_datetime_tz
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts0]
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts1]
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_is_utc[ts2]
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range0]
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range1]
> FAILED pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_tz_range_is_utc[tz_range2]
>
> produce DeprecationWarning: an integer is required (got type float).
> Implicit conversion to integers using __int__ is deprecated, and may be
> removed in a future version of Python.
> I filtered this warning for now.
> I'll open separate issues for those two warnings after this PR is merged.
>
> @VirosaLi <https://github.com/VirosaLi> thanks.
> so looks all good. what is the status of the 32-bit build?
>
> It failed. So I left it commented. We could open another issue for the
> 32-bit build.
>
> do you recall how much work to get it running?
>
> ok on opening an issue, can you followup on that build?
>
> I'm not familiar with the 32-bit build. If I remember it correctly, it has
> been commented out for a while. I could work on it later this week.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/35214#issuecomment-671657867>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIVHLXOQ7JX7XO5L55TSACHB3ANCNFSM4OWVSJYQ>
> .
>

thanks @VirosaLi very nice! i know this was a lot of effort

if you wouldn't mind opening the issues would be great