CI: Move sql builds from Travis to Github actions
State: closed

try to move builds with databases to GitHub actions


Labels:
name: CI
description: Continuous Integration

Comments:
marking this as 1.2 as any changes to the CI will also need to be backported. but not a blocker for 1.2.
looks good, @fangchenli can you make the change indicated by @simonjayhawkins merge master and ping on green (possible numpy-dev is still failing but that's ok)
I suspect SQL related tests did not run in this build. Let me double check.
The py38-slow build is not running any SQL tests. Doesn't it suppose to? If not, we could move it to Azure.
> The py38-slow build is not running any SQL tests. Doesn't it suppose to? If not, we could move it to Azure.

umm, whereever we have sqlalchemy installed should run (if it ins't we should add)
i think the npdev build is fixed on master if you can merge master
> > The py38-slow build is not running any SQL tests. Doesn't it suppose to? If not, we could move it to Azure.
> 
> umm, whereever we have sqlalchemy installed should run (if it ins't we should add)

Sqlalchemy is installed. I've commented out all SQL setup code and removed sqlalchemy. And the number of tests run is the same as before. So I think none of the SQL tests were run on this build.
interesting

ahh this a slow build - thought they were only slow tests

ok as long as we have sufficient coverage on sal tests its fine (and can revisit that later anyhow)

if u would open an issue
@fangchenli CI is currently failing on master (for other reasons). but ideally like to get this in. can you summarize what this is accomplishing:

- moving all database builds to azure
- enabling sql tests on 3.8_slow?
- removing builds from travis? 
what is left after this PR
@jreback The goal of this PR is to move all database builds to GitHub actions. After this, what left in Travis is the 3.8_slow #38429 and arm64.
And I'll have time to work on this after tomorrow.
same on this one. will be putting in 1.2.x
ping when ready with this one
and if you can merge master on this one
@fangchenli status of this one?
> @fangchenli status of this one?

Still working on it. It's harder than I thought.
> Still working on it. It's harder than I thought.

moving to 1.2.1
The SQL databases are working correctly. Two issues remain.

1. We are currently testing the minimum support version of several packages in the locale build.

    - blosc=1.15.0
    - fastparquet=0.3.2
    - xarray=0.12.3
    - psycopg2=2.7
    - pymysql=0.7.11
    - sqlalchemy=1.3.0

They are causing some conflicts, and conda can not resolve them. It's failing on Travis too. I unpinned all of them for now.

2.  There are lots of failed tests and `TypeError: '<' not supported between instances of 'str' and 'int'` (some are from pyarrow and xarray). I'm not sure what is causing this many errors. They are not sql-related.


ok see if u can find the min pins to move these pins

likely the errors are related to other deps moving
you can try looking back in the build logs on master and see when this job first started failing - compare the before env and maybe can see what changes (and what we need to pin)

alt can try to create the env localky - conda has some verbose options to let u know what may is in conflict 
> you can try looking back in the build logs on master and see when this job first started failing - compare the before env and maybe can see what changes (and what we need to pin)
> 
> alt can try to create the env localky - conda has some verbose options to let u know what may is in conflict

The conflict is caused by blosc. Increasing its minimum version to 1.17.0 fixed it.
Some of the errors look like #30333.
was the numpy version bumped?

also ok to omit blosc entirely
> was the numpy version bumped?
> 
> also ok to omit blosc entirely

numpy version was not specified in this build.
Using checkout v1 fixed most of those errors and test failures. But some remain

```bash
=========================== short test summary info ============================
FAILED pandas/tests/io/test_user_agent.py::test_server_and_default_headers[ParquetFastParquetUserAgentResponder-read_parquet-34273-fastparquet]
FAILED pandas/tests/io/test_user_agent.py::test_server_and_custom_headers[ParquetFastParquetUserAgentResponder-read_parquet-34275-fastparquet]
FAILED pandas/tests/io/test_user_agent.py::test_to_parquet_to_disk_with_storage_options[fastparquet]
FAILED pandas/tests/util/test_show_versions.py::test_show_versions - Attribut...
ERROR pandas/tests/io/test_fsspec.py - AttributeError: module 'numba' has no ...
ERROR pandas/tests/io/test_fsspec.py - AttributeError: module 'numba' has no ...
ERROR pandas/tests/io/test_gcs.py - AttributeError: module 'numba' has no att...
ERROR pandas/tests/io/test_gcs.py - AttributeError: module 'numba' has no att...
ERROR pandas/tests/io/test_parquet.py - AttributeError: module 'numba' has no...
ERROR pandas/tests/io/test_parquet.py - AttributeError: module 'numba' has no...
= 4 failed, 146993 passed, 3499 skipped, 1084 xfailed, 63 xpassed, 7 warnings, 6 errors in 1331.13s (0:22:11) =
```
can u merge master
All those errors are related to https://github.com/dask/fastparquet/issues/539. We need to either bump min fastparquet version to 0.4.0 or pin numba version to <0.52
> All those errors are related to [dask/fastparquet#539](https://github.com/dask/fastparquet/issues/539). We need to either bump min fastparquet version to 0.4.0 or pin numba version to <0.52

let's bump fastparquet & add a note in the whatsnew (do in 1.2.1 is fine)
thanks @fangchenli really monumental here. !

a couple of followon things if you can.

- add the build versions printing for these 2 builds
- for the -cov build. i know we are attempting to get coverage here, but anything we can remove / reduce here? e.g. it would make sense to split this into 2 builds one for slow and one for the fast. though we *already* have these covered (except for the slow / db things). so maybe can just call this a slow build and repurpose one of the azure builds (38 linux maybe) as a coverage build?
@meeseeksdev backport 1.2.x