Repository: kivy/python-for-android

Issue:
Numpy support w/ python3crystax
I'm trying to get my app to compile with Python 3 using the Crystax NDK.  Everything is set up, and if I try to push my app to my device without specifying that numpy is needed, everything works fine.  However, numpy raises a few issues:
1. (easily fixed) The recipe needs to be modified such that python3crystax is supported.  Currently, it only claims python2 support, but that's overridden easily enough (`depends = ['python2']` -> `depends = [('python3crystax', 'python2)]` in the `__init__.py`).
2. The compiler claims it can't find -lcrystax:

``` sh
...
don't know how to compile Fortran code on platform 'posix'
C compiler: /usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -mandroid -fomit-frame-pointer --sysroot /home/wwoods/walt/dev/kivy/crystax-ndk-10.3.2/platforms/android-19/arch-arm -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DANDROID -mandroid -fomit-frame-pointer --sysroot /home/wwoods/walt/dev/kivy/crystax-ndk-10.3.2/platforms/android-19/arch-arm -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/home/wwoods/miniconda3/envs/kivy/include/python3.5m -c'
ccache: _configtest.c
/usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -mandroid -fomit-frame-pointer --sysroot /home/wwoods/walt/dev/kivy/crystax-ndk-10.3.2/platforms/android-19/arch-arm _configtest.o -o _configtest
/home/wwoods/walt/dev/kivy/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.3/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lcrystax
collect2: error: ld returned 1 exit status
/home/wwoods/walt/dev/kivy/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.3/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lcrystax
collect2: error: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o
```

Everything except numpy works fine, so I think that it's just Numpy's recipe that needs a minor alteration.  Oddly though, specifying a LIBRARY_PATH or LDFLAGS does not have any effect.  In fact, if I add a `get_recipe_env` to the `NumpyRecipe` that prints out what `super().get_recipe_env(arch)` contains, it has LDFLAGS with the appropriate path on there.  Running the linker by hand with the given flags find the library fine (`ld $(LDFLAGS) -lcrystax --verbose`).  In the context of python-for-android though, the linker always fails to find libcrystax.a.

Any help appreciated, thanks.



Discussion:
PS - I'm using kivy1.9.2dev and buildozer android_new as the frontend to python-for-android.

Sorry for the delay replying, I did think about this but I'm not sure offhand what could be wrong. I'll try to check it out when I get back around to some of the python3 things.

I am guessing that since this issue is still open, the Numpy recipe will not support python3crystax easily?
I don't know if it will be actually hard to make it work, but yes, there
has been no further development of this issue.

On 25/02/17 08:13, johnbolia wrote:
> I am guessing that since this issue is still open, the Numpy recipe will
> not support python3crystax easily?
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/kivy/python-for-android/issues/882#issuecomment-282468928>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABNQmxvCzzwK3b8hI7Kh2uhlBPpCMJcDks5rf-KsgaJpZM4J5kQN>.
> 



Isn't it also an issue that the Crystax NDK currently doesn't ship a Fortran compiler for arm-linux-androideabi? Other than that it should be possible to set something like
```
setup_extra_args = ['-L', 'CRYSTAX/sources/crystax/libs/armeabi-v7a']
```
in the numpy recipe to pass additional flags to its build_ext.
@renemilk Fortunately numpy doesn't require any fortran compiling. This is a barrier for scipy though.
I tried passing the extra libdir via the mentioned build_ext flag, but it still does not show up in the failing linker command. Maybe putting a ```site.cfg``` in the extracted numpy sources can help, but I'm not hopeful. The nuclear option would be substituting the linker with a wrapper script that forcibly adds the flag unconditionally.
@renemilk You're trying to make the numpy build link properly with libcrystax.so? It may also be plausible to try to remove this linking as I don't think libcrystax should be necessary. I've tried both, also without success.
Yes, I do. 
I have no idea idea how, why or when libcrystax gets injected into the linker command there, but yeah I also think removing it would do. 
Edit, I got it working for crystax 2.7 here https://github.com/frmdstryr/p4a-numpy . Have not tested on 3.5 but I believe it should update everything to the correct versions.
Wow, great! I was going to say that your post was very helpful, but getting it all working is even better :p .

Also, I've tried crystax python 2.7 before, but hit some issues with it (I don't remember what). Have you got that working with p4a too?
Yes python2crystax works fine  (been using it for 6+ months now). For some reason it is mentioned that `Py_SetPath` is unsupported in https://github.com/kivy/python-for-android/blob/master/pythonforandroid/bootstraps/sdl2/build/jni/src/start.c#L131 however `Py_SetPath` works fine with the crystax version (however the python2 version does NOT have Py_SetPath!).

Also have a python2crystax recipe with openssl for that here https://github.com/frmdstryr/p4a-crystax (based on a PR someone did a while ago)

Edit: Changes to p4a might be required for it though I have my own fork
Hi,
I'm trying to make numpy work with py3crystax too (internship in a lab and they need numpy on android with kivy and python3) so I'll work on it for some time.

If I get interesting results I'll communicate them here, also if you don't have time to do some tests but have ideas about what to do, share them here.
If I have enough time I'll try your suggestions and post the results.
Sharing my discoveries, or at least the ones I think as important:
After editing the recipe for numpy with depends python3crystax only I get the same error as there : #1074 
I can set `[('python3crystax','python3')]` or `[('python3crystax','python2')]` or `['python3crystax']` I get the same error. (Beware of the order! Setting crystax after python2/3 gives /usr/bin/false command not found)

The problem usually starts with :
```
blas_mkl_info:
  libraries  not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

openblas_info:
  libraries  not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries  not found in ['/usr/local/lib', '/usr/lib/atlas-base', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE
etc.....
```
If I'm not mistaking the thread says the error has been resolved at that time by editing numpy recipe. There were no flags that's why numpy gave this error and since I have the same error with python3 I guess the libraries are not correctly set.
To be sure I've made another project to test it: python2 kivy numpy, ndk and sdk already downloaded (also tested by setting nothing, buildozer downloads the good one) and it works fine.

So the only difference I have with the working project is python3, more exactly python3crystax in the ndk crystax 10.3.2.
The libraries added in the recipe refer to this with arch arm: 
```
flags = " -L{ctx.ndk_dir}/platforms/android-{api_ver}/arch-{platform}/usr/lib/" \
                " --sysroot={ctx.ndk_dir}/platforms/android-{api_ver}/arch-{platform}" \
```
I've compared the content between crystax ndk and google r9c ndk.
Some files are missing or added in android-19/arch-arm/usr/lib but that's not what surprised me. Every lib****.a in the crystax version are empty, only 8 bytes!
With an hex editor we can see there is only `!<arch>.`
I tried to copy paste the content of usr from the r9c ndk (hey don't hit me, I wanted to try) but no changes I get the same error even after a buildozer android clean.

I tried to edit the flags following those instructions : https://github.com/xianyi/OpenBLAS/wiki/How-to-build-OpenBLAS-for-Android
```
        if flags not in env['CC']:
            env['CC'] += flags + ' NOFORTRAN=1 -mhard-float -D_NDK_MATH_NO_SOFTFP=1'
        if flags not in env['LD']:
            env['LD'] += flags + '-Wl,--no-warn-mismatch -lm_hard' +' -shared'
```

but I think I did something wrong and I don't have enough time today to continue I'll see this later.
Other important point:
I followed instructions to ignore the blas lapack atlas inclusion, which is possible but we loose a lot of speed with algebra computation (see https://docs.scipy.org/doc/numpy-1.9.2/user/install.html and https://stackoverflow.com/questions/32228967/compile-numpy-without-intel-mkl-blas-atlas-lapack). It should work but I still get the same last error and I think it's this one that halts everything:
```
C compiler: /usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -mandroid -fomit-frame-pointer -D__ANDROID_API__=19 -I/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -isysroot /media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -I/media/axel/reste/buildozer/.buildozer/android/platform/build/build/python-installs/medmanip/include/python3.5 -L/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm/usr/lib/ --sysroot=/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -DANDROID -mandroid -fomit-frame-pointer -D__ANDROID_API__=19 -I/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -isysroot /media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -I/media/axel/reste/buildozer/.buildozer/android/platform/build/build/python-installs/medmanip/include/python3.5 -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python3.5m -c'
ccache: _configtest.c
/usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -mandroid -fomit-frame-pointer -D__ANDROID_API__=19 -I/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -isysroot /media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -I/media/axel/reste/buildozer/.buildozer/android/platform/build/build/python-installs/medmanip/include/python3.5 -L/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm/usr/lib/ --sysroot=/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm _configtest.o -o _configtest
/media/axel/reste/.buildozer/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.3/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lcrystax
collect2: error: ld returned 1 exit status
/media/axel/reste/.buildozer/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.3/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lcrystax
collect2: error: ld returned 1 exit status
failure.
```
lcrystax is missing!
In #1074 it was resolved by adding the libraries (seems like it works with python2crystax) but with the exact same path now it gives the same error they had before. So I keep thinking there is a problem with python3crystax, I would prefer there isn't because I tried to pick the content or android-19/arch-arm in there latest version (b1052) and the .a libs are still empty...

N.B: The warnings saying lapack, blas and atlas are missing don't fail the compilation. The failure is due to crystax missing.


I resolved this error by editing the flags in numpy recipe adding `-L{ctx.ndk_dir}/sources/crystax/libs/{arch.arch}` the the compilation started to work and again failed with an error over python.h.
After searching for some time I found that it was trying to compile my python3.5 installed in my computer rather than crystax's one.
I don't know why it did that but I edited the flags once more and it resolved this problem `-I{ctx.ndk_dir}/sources/python/{py_ver}/include/python/`.
By the way, after looking at the flags there https://github.com/frmdstryr/p4a-numpy/blob/master/p4a_numpy/__init__.py (it is what was used to edit the broken recipe of numpy for the py2.7 version last January or December), I saw that it had the correct flags to avoid this so I copy pasted all their flags.

Now the new problem is when building the extensions:
```
ccache: numpy/core/src/dummymodule.c
[.........]
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/numpy
creating build/lib.linux-x86_64-3.5/numpy/core
```
It's trying to build a library but I get this error:
`........ crystax-ndk-10.3.2/platforms/android-19/arch-arm/usr/lib/../lib/crtbegin_so.o: Relocations in generic ELF (EM: 40)`
After searching on internet I found that it's either due to using the wrong crtbegin, by wrong I mean one that has not been compiled with the same compilator, this one is in arch-arm so I hope it's good, or it could be because the compilator doesn't know how to use it and it's resolved with flags (good, until know everything has been resolved with flags).

Now I'm stuck, I don't know what to do because CC and CXX already have what I intended to put : `/usr/bin/ccache arm-linux-androideabi-gcc` (and g++ respectively) 
Unfortunately I don't have anymore time to work on this but I hope my explanations will save a lot of time to the next person trying to resolve this problem.

Here the last error giving me a headache, I'm not used to compilation problems but isn't it strange that it uses linux x86_64 to compile for arm? Until now the commands visible in the console started with `C compiler: /usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -mandroid -fomit-frame-poi`

`error: Command "x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro --sysroot /media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -lm -L/media/axel/reste/buildozer/.buildozer/android/platform/build/build/libs_collections/medmanip/armeabi-v7a -L/media/axel/reste/.buildozer/crystax-ndk-10.3.2/sources/crystax/libs/armeabi-v7a -lcrystax -DANDROID -mandroid -fomit-frame-pointer -D__ANDROID_API__=19 -I/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -isysroot /media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -I/media/axel/reste/buildozer/.buildozer/android/platform/build/build/python-installs/medmanip/include/python3.5 -march=armv7-a -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/numpy/core/src/dummymodule.o -Lbuild/temp.linux-x86_64-3.5 -o build/lib.linux-x86_64-3.5/numpy/core/_dummy.cpython-35m-x86_64-linux-gnu.so" failed with exit status 1`
Did anyone have any luck with this?
Hi guys, I think I figured it out!

Here is the fixed recipe with a minimal example application: https://github.com/plapadoo/p4a-numpy

This isn't well-tested. So I will collect more feedback before creating a PR. Please let me know if the fixed recipe works for you.

Thanks to all the people who already put some effort into this in the past. Especially @Abysselene and @frmdstryr 
Fantastic, I will try it out and let you know how I get on.

Thank you very much!

On Wed, 8 Aug 2018 18:49 Kim Rinnewitz, <notifications@github.com> wrote:

> Hi guys, I think I figured it out!
>
> Here is the fixed recipe with a minimal example application:
> https://github.com/plapadoo/p4a-numpy
>
> This isn't well-tested. So I will collect more feedback before creating a
> PR. Please let me know if the fixed recipe works for you.
>
> Thanks to all the people who already put some effort into this in the
> past. Especially @Abysselene <https://github.com/Abysselene> and
> @frmdstryr <https://github.com/frmdstryr>
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/kivy/python-for-android/issues/882#issuecomment-411493567>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACT71o16DXs1QTgcVIl1RdtuQ2hoAoJxks5uOySNgaJpZM4J5kQN>
> .
>


Resolving patch:
diff --git a/pythonforandroid/recipes/numpy/__init__.py b/pythonforandroid/recipes/numpy/__init__.py
--- a/pythonforandroid/recipes/numpy/__init__.py
+++ b/pythonforandroid/recipes/numpy/__init__.py
@@ -1,48 +1,48 @@
 from pythonforandroid.recipe import CompiledComponentsPythonRecipe
 from pythonforandroid.toolchain import warning
+from os.path import join
 
 
 class NumpyRecipe(CompiledComponentsPythonRecipe):
 
-    version = '1.9.2'
-    url = 'https://pypi.python.org/packages/source/n/numpy/numpy-{version}.tar.gz'
-    site_packages_name= 'numpy'
+    version = '1.15.1'
+    url = 'https://pypi.python.org/packages/source/n/numpy/numpy-{version}.zip'
+    site_packages_name = 'numpy'
 
-    depends = ['python2']
+    depends = [('python2', 'python3crystax')]
 
-    patches = ['patches/fix-numpy.patch',
-               'patches/prevent_libs_check.patch',
-               'patches/ar.patch',
-               'patches/lib.patch']
+    patches = [
+        join('patches', 'fix-numpy.patch'),
+        join('patches', 'prevent_libs_check.patch'),
+        join('patches', 'ar.patch'),
+        join('patches', 'lib.patch'),
+        join('patches', 'python2-fixes.patch')
+    ]
 
     def get_recipe_env(self, arch):
-        """ looks like numpy has no proper -L flags. Code copied and adapted from
-            https://github.com/frmdstryr/p4a-numpy/
-        """
-
         env = super(NumpyRecipe, self).get_recipe_env(arch)
-        #: Hack add path L to crystax as a CFLAG
-
-        py_ver = '3.5'
-        if {'python2crystax', 'python2'} & set(self.ctx.recipe_build_order):
-            py_ver = '2.7'
 
-        py_so = '2.7' if py_ver == '2.7' else '3.5m'
+        flags = " -L{} --sysroot={}".format(
+            join(self.ctx.ndk_platform, 'usr', 'lib'),
+            self.ctx.ndk_platform
+        )
+
+        if self.ctx.ndk == 'crystax':
+            py_ver = self.ctx.python_recipe.version[0:3]
+            src_dir = join(self.ctx.ndk_dir, 'sources')
+            py_inc_dir = join(src_dir, 'python', py_ver, 'include', 'python')
+            py_lib_dir = join(src_dir, 'python', py_ver, 'libs', arch.arch)
+            cry_inc_dir = join(src_dir, 'crystax', 'include')
+            cry_lib_dir = join(src_dir, 'crystax', 'libs', arch.arch)
+            flags += ' -I{}'.format(py_inc_dir)
+            flags += ' -L{} -lpython{}m'.format(py_lib_dir, py_ver)
+            flags += " -I{}".format(cry_inc_dir)
+            flags += " -L{}".format(cry_lib_dir)
 
-        api_ver = self.ctx.android_api
-
-        platform = 'arm' if 'arm' in arch.arch else arch.arch
-        #: Not sure why but we have to inject these into the CC and LD env's for it to
-        #: use the correct arguments.
-        flags = " -L{ctx.ndk_dir}/platforms/android-{api_ver}/arch-{platform}/usr/lib/" \
-                " --sysroot={ctx.ndk_dir}/platforms/android-{api_ver}/arch-{platform}" \
-            .format(ctx=self.ctx, arch=arch, platform=platform, api_ver=api_ver,
-                    py_so=py_so, py_ver=py_ver)
         if flags not in env['CC']:
             env['CC'] += flags
         if flags not in env['LD']:
             env['LD'] += flags + ' -shared'
-
         return env
 
     def prebuild_arch(self, arch):


Test patch: