CI: Adding build for ARM64
State: closed

Added arm64 test support in travis-ci .
Modified environment creation by using archiconda instead of miniconda as miniconda is not supported in arm64 currently .

- [X] closes https://github.com/pandas-dev/pandas/issues/28986
- [X] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
@ossdev07 still active? If so can you fix merge conflicts and address comments?
Hi @WillAyd , I am working on it. I will let you know once I am done.
Hi @WillAyd and @datapythonista,

-  I have removed archiconda and updated it to miniforge. It does not require sudo permissions to run conda commands as it is similar to miniconda. By this we can remove all the unwanted code as suggested previously.

-  For arm64 xvfb need to be installed seperately, so added in apt-addons in travis.

-  Flags "-n auto --dist=loadfile -s --strict" mentioned in pytest command in run_tests.sh are giving timeout issue in arm64 platform. So, I have removed the same for arm64 platform.

Please review and provide feedback if any more updates are required.
@datapythonista , 

Sorry, for the extra commits that have been added. Rebase has gone wrong and all commits got added. The actual commit which needs to be committed is https://github.com/pandas-dev/pandas/pull/30641/commits/887591e8a5a529ef250d3a67364280008b0e14c5. 
> Sorry, for the extra commits that have been added. Rebase has gone wrong and all commits got added. The actual commit which needs to be committed is [887591e](https://github.com/pandas-dev/pandas/commit/887591e8a5a529ef250d3a67364280008b0e14c5).

You'll have to fix your branch for us to merge this. I wrote some time ago on what I usually do in these cases in this post: https://datapythonista.me/blog/useful-git-commands.html

Hope that helps.
> > Sorry, for the extra commits that have been added. Rebase has gone wrong and all commits got added. The actual commit which needs to be committed is [887591e](https://github.com/pandas-dev/pandas/commit/887591e8a5a529ef250d3a67364280008b0e14c5).
> 
> You'll have to fix your branch for us to merge this. I wrote some time ago on what I usually do in these cases in this post: https://datapythonista.me/blog/useful-git-commands.html
> 
> Hope that helps.

Thanks @datapythonista. I have resolved the conflicts in the commits and added the commit with changes suggested. Please review and provide feedback if anything needs to be updated.

(I work with @ossdev07 ) : @ossdev07 as an independent task, can you make a note of the missing `xvfb` for Travis-CI in their community forums: https://travis-ci.community/
> (I work with @ossdev07 ) : @ossdev07 as an independent task, can you make a note of the missing `xvfb` for Travis-CI in their community forums: https://travis-ci.community/

@rhenwood-arm ,

I will take a note of it and will raise it in the travis community.
Travis is red, can you have a look and see what's wrong please
> Travis is red, can you have a look and see what's wrong please

@datapythonista ,

After adding all flags in pytest command, it is failing with the timeout issue at different steps for each build for arm64 platform. That is the reason I have removed those three flags.

Could you please suggest if there is any resolution to run without these flags for a specific platform?
IIRC the flags you changed were for `pytest`, and in this case `pytest` didn't even started, so that's surely not going to solve the problem.

The build was stalled during `ci/submit_cython_cache.sh`. Can you merge master, so we can see if the problem happens at the same stage. I'm wondering if Travis is reliable for ARM64...
> IIRC the flags you changed were for `pytest`, and in this case `pytest` didn't even started, so that's surely not going to solve the problem.
> 
> The build was stalled during `ci/submit_cython_cache.sh`. Can you merge master, so we can see if the problem happens at the same stage. I'm wondering if Travis is reliable for ARM64...

@datapythonista , 

After merging master build is stalled during `ci/submit_cython_cache.sh` in travis check in this PR, whereas in personal travis, it is getting stalled during `pytest`. Please check report https://travis-ci.org/github/ossdev07/pandas/jobs/664267609 for the same. 

Can you please suggest if I am missing anything? Thanks.

Seems like a problem in Travis to me.

May be you can comment the two lines about the cache in `travis.yml`, and see if that works:
```
  # - ci/prep_cython_cache.sh
  - ci/setup_env.sh
  # - ci/submit_cython_cache.sh
```
May be that helps us understand better the problem. But I think you'll have to report it to Travis, doesn't seem to be a problem in our side.
> Seems like a problem in Travis to me.
> 
> May be you can comment the two lines about the cache in `travis.yml`, and see if that works:
> 
> ```
>   # - ci/prep_cython_cache.sh
>   - ci/setup_env.sh
>   # - ci/submit_cython_cache.sh
> ```
> 
> May be that helps us understand better the problem. But I think you'll have to report it to Travis, doesn't seem to be a problem in our side.

I have commented cache part and checked in local travis. It is getting timed out during pytest phase. 
Report link: https://travis-ci.org/github/ossdev07/pandas/jobs/664711811.




Can you report the problem to TravisCI? I don't see what we can do in our side if their systems stall during the build in a somehow random way. 
> Can you report the problem to TravisCI? I don't see what we can do in our side if their systems stall during the build in a somehow random way.

Thanks for the suggestion. I will raise an issue in travis community regarding the same and update if anything is required.

Hi @datapythonista ,

I have raised a ticket https://travis-ci.community/t/travis-build-gets-stalled-for-arm64-job-in-pandas-package/7678 in travis community for the build issue. 

As suggested in that ticket, I have rebased and added travis_wait to increase the timeout before each script. But I was facing pandas compilation issue.

**Error:**
`pandas/_libs/src/ujson/lib/ultrajsonenc.c:726:59: error: ‘fastcall’ attribute directive ignored [-Werror=attributes]
                                                           char *end) {
                                                           ^
cc1: all warnings being treated as errors
gcc -pthread -shared -B /home/travis/miniconda3/envs/pandas-dev/compiler_compat -L/home/travis/miniconda3/envs/pandas-dev/lib -Wl,-rpath=/home/travis/miniconda3/envs/pandas-dev/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-aarch64-3.7/pandas/io/sas/sas.o -o /home/travis/build/ossdev07/pandas/pandas/io/sas/_sas.cpython-37m-aarch64-linux-gnu.so`

I have tried after removing travis_wait as well but still facing the same issue. Please check the report https://travis-ci.org/github/ossdev07/pandas/jobs/671601371 for the same.

Can you suggest any pointers on resolving this issue?
No idea to be honest, but looks like an issue specific to ARM64.

I don't think the `travis_wait` will fix the problems, but let's try.
@datapythonista ,

The compilation issue is resolved when we remove "-Werror" flag in extra_compile_args variable in setup.py. But added travis_wait for running tests, but still facing the same timeout issue. If we remove the previously mentioned flags in pytest command, it is working successfully. Can you please suggest what I may be missing?

Report: https://travis-ci.org/github/ossdev07/pandas/jobs/674356951
If it works without `-Werror` it means that the error is a warning. But we don't allow warnings in the build anyway.

But I think the problem here is the timeout. Not sure if it's a problem with Travis ARM64 machines that they stall, or if they are just too slow to compile, run the tests...
@datapythonista ,

I have tried running locally again and tests are triggered successfully:

**xvfb-run pytest -m "(not slow and not network and not clipboard)" -n auto --dist=loadfile -s --strict --durations=10 --junitxml=test-data.xml pandas**
===================== test session starts ==============================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.12.0
rootdir: /pruthvi/pandas, inifile: setup.cfg, testpaths: pandas
plugins: hypothesis-5.8.3, forked-1.1.3, xdist-1.31.0
gw0 [68456] / gw1 [68456] / gw2 [68456] / gw3 [68456]
=4 failed, 63509 passed, 3907 skipped, 1044 xfailed, 7 xpassed, 29 warnings in 1357.02s (0:22:37) =

I think, in travis issue is with xvfb. It is not getting started.
**sudo systemctl start xvfb
Failed to start xvfb.service: Unit xvfb.service not found.**

In travis test run is triggered without xvfb-run:
**pytest -m "(not slow and not network and not clipboard)" -n auto --dist=loadfile -s --strict --durations=10 --junitxml=test-data.xml pandas**
=== test session starts ====
platform linux -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.12.0
rootdir: /home/travis/build/ossdev07/pandas, inifile: setup.cfg, testpaths: pandas
plugins: hypothesis-5.8.3, forked-1.1.3, xdist-1.31.0
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I

I suppose it is getting timeout due to this reason. I have added xvfb as services but still getting issue in starting xvfb services.

Can you suggest on the same? 
I don't think `xvfb` is the problem, but I may be wrong. Let me explain about `xvfb`.

pandas is able to get and set data from and to the system clipboard. In Windows that is done "directly", but Linux doesn't have a clipboard itself, but it's the X server that usually provides it. In a desktop machine, if you're using pandas locally with your browser, you'll have an X server running. But in the CI (or when using pandas in a server), the Linux doesn't have more than the terminal, so no X server exists. `xvfb` is a "simulator" of the X server. So, when we call the tests, we wrap the call with `xvfb-run`, so the clipboard tests have an X server to use, and tests don't fail.

No idea on whether arm64 uses X server, or what. But you can control when we use `xvfb` with this condition: https://github.com/pandas-dev/pandas/blob/master/ci/run_tests.sh#L18

Also, we're able to control whether we run the clipboard tests in a build using `not clipboard`, like in here: https://github.com/pandas-dev/pandas/blob/master/ci/azure/posix.yml#L21

So, if what you need, is to not use `xvfb`, that's fine, and you can skip the clipboard tests in arm if they don't work. But I think the problem is more that the Travis hardware is just too slow, and it's cancelled after some step takes more than 10 minutes without returning any output.
@datapythonista ,

I have added travis_wait and installed xvfb. Tests are getting exited with below error:

xvfb-run pytest -m "(not slow and not network and not clipboard)" -n auto --dist=loadfile -s --strict --durations=10 --junitxml=test-data.xml pandas
===== test session starts ================
platform linux -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.12.0
rootdir: /home/travis/build/ossdev07/pandas, inifile: setup.cfg, testpaths: pandas
plugins: hypothesis-5.8.3, forked-1.1.3, xdist-1.31.0
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I
[gw14] node down: Not properly terminated
replacing crashed worker gw14
[gw1] node down: Not properly terminated
replacing crashed worker gw1
[gw7] node down: Not properly terminated
replacing crashed worker gw7
[gw11] node down: Not properly terminated
replacing crashed worker gw11
[gw6] node down: Not properly terminated
replacing crashed worker gw6
[gw13] node down: Not properly terminated
replacing crashed worker gw13
The command "travis_wait 30 ci/run_tests.sh" exited with 137.

**Report:** https://travis-ci.org/github/ossdev07/pandas/jobs/683298787

I suppose it is giving error with "-n auto" flag. Could you provide me any suggestions on the above issue.
Thanks @ossdev07 

For what I've seen, there is a discussion now on whether we want to support new architectures or not in #33971. You may want to have a look there, not sure if it's worth continuing with this until a decision is made there.

For the error, seems again a problem with Travis infrastructure. Not sure why the testing process is crashing, but I guess it's the OS killing it because of lack of resources, or something like that. Not sure if removing the `-n auto` will solve the problem, but can provide more information (the log is not so verbose when tests are running in parallel).
FWIW, I don't think https://github.com/pandas-dev/pandas/issues/33971 necessarily should hold this PR up. I think whatever rules we establish, ARM64 will qualify for inclusion.
@datapythonista ,

> For the error, seems again a problem with Travis infrastructure. Not sure why the testing process is crashing, but I guess it's the OS killing it because of lack of resources, or something like that. Not sure if removing the `-n auto` will solve the problem, but can provide more information (the log is not so verbose when tests are running in parallel).

I have verified with parallel flag as "-n 4" instead of auto and without travis_wait, tests are triggering and passing in both amd64 and arm64 jobs.
I have just modified at line https://github.com/ossdev07/pandas/blob/arm64_test_sample/ci/run_tests.sh#L23

**Travis Report:** https://travis-ci.org/github/ossdev07/pandas/jobs/683704969

Can we proceed with these changes or flag must be auto only? 
That's interesting. I don't see those changes in this PR. Did you revert them, or tried somewhere else? I'm in my phone, so may be I'm just not seeing them.

The auto means to use and many workers as cores.

If you don't mind, there are two things I'd like to do:

We can add in the yaml a variable for the number of workers (e. g. `PYTEST_WORKERS`). Just set it to `4` for the arm build. Then in the test script, add an `if`, and set it to `auto`, if the variable is not set. This should make the arm build pass, without changing the workers in the other builds.

In parallel, would be good to find out how many workers `auto` means in arm. I guess it can be more than 8, and the workers crash because of memory (the more the workers, the more memory used, and if we run out of memory, the workers will crash). If you can also check the amount of memory in arm, and in Linux for comparison, that would be great. Understanding this will be very helpful. We want to optimise the number of workers, so the build doesn't take too long. This is taking 45 minutes, and it's by far the slowest build. Which means that for any PR, we'll have to wait for the arm build before being able to merge, if we merge the way you propose. And also, we want to understand the memory, we don't want to merge this being in the edge, and that the next PR adds a test, and this starts failing because of lack of memory.

Thanks for the work on this, and sorry it's taking so long. 
@datapythonista ,

> We can add in the yaml a variable for the number of workers (e. g. `PYTEST_WORKERS`). Just set it to `4` for the arm build. Then in the test script, add an `if`, and set it to `auto`, if the variable is not set. This should make the arm build pass, without changing the workers in the other builds.
> 

- I have added PYTEST_WORKERS as a global variable with value "auto" and reassigning it in arm64 job. I have tested with -n value as 12 and 16 as well but workers are getting crashed and it is working with 8 workers.
- I have added xvfb installation separately only for arm64 as it is not getting installed as services. 
- Flag "-Werror" had to be removed as compilation of external dependencies were failing with warnings treated as error. 

Please check the add commit and provide feedback if anything needs to be changed.
these changes are breaking other builds: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=35039&view=logs&j=bef1c175-2c1b-51ae-044a-2437c76fc339&t=770e7bb1-09f5-5ebf-b63b-578d2906aac9&l=17


Do the workarounds described in https://github.com/pandas-dev/pandas/issues/17792 need to be incorporated in order for the test suite to pass?