Repository: numpy/numpy

Issue:
BLD: hardcoded SSE4.2 for Intel compilers on Linux
Building numpy using Intel compilers on Linux systems still breaks on CPUs that does not support SSE4.2.

I would suggest using `-xHOST` flag instead of `-xSSE4.2` (and on windows `/QxHOST` insted of `/QaxSSE4.2`)

For the Linux part, my suggested patch is:

```
--- ./numpy/distutils/intelccompiler.py.orig    2016-01-06 22:07:54.000000000 +0100
+++ ./numpy/distutils/intelccompiler.py 2016-02-19 15:07:00.438235792 +0100
@@ -54,7 +54,7 @@
     def __init__(self, verbose=0, dry_run=0, force=0):
         UnixCCompiler.__init__(self, verbose, dry_run, force)
         self.cc_exe = ('icc -m64 -fPIC -fp-model strict -O3 '
-                       '-fomit-frame-pointer -openmp -xSSE4.2')
+                       '-fomit-frame-pointer -openmp -xHOST')
         compiler = self.cc_exe
         if platform.system() == 'Darwin':
             shared_flag = '-Wl,-undefined,dynamic_lookup'
--- ./numpy/distutils/fcompiler/intel.py.orig   2016-01-07 03:16:25.000000000 +0100
+++ ./numpy/distutils/fcompiler/intel.py        2016-02-19 15:07:16.424220745 +0100
@@ -123,7 +123,7 @@
         return ['-openmp -fp-model strict']

     def get_flags_arch(self):
-        return ['-xSSE4.2']
+        return ['-xHOST']

 # Is there no difference in the version string between the above compilers
 # and the Visual compilers?
```



Discussion:
Maybe a PR? Also, what version of Numpy, Linux, Linux Flavor, etc. are you using?

was added recently in gh-6243
@charris I don't see a reason, was it just an oversight?
sse4.2 by default should a be no-go

gh-6243 was submitted by an @dzagorny, an Intel engineer. Let's CC them and see if they have any comments :-)

@jakirkham @njsmith 
I was using `Numpy 1.10.4`, `Scientific Linux 6.6`
Anyway, the -xSSE4.2 will generate a code that will run only on SSE4.2 enabled CPUs. So here is the CPU is important, not the system. The compilation will finish without any error message, but the code will fail with an illegal instruction (core dump) on non SSE4.2 capable CPUs. 
Additionally, this will restrict the code to SSE4.2 on newer CPUs, so no AVX for example.

@hajgato: Yeah, the current behavior is clearly wrong. But none of the numpy maintainers use `icc` and Intel mostly maintains the numpy `icc` support, so I'm sort of hoping they'll have some recommendation on what to use instead. (`-xHOST` creates binaries that might not work if moved to new systems, right? Is there some alternative that makes a better default? Beats me :-))

CC: @cgohlke @ilanschnell @cournape for some other folks that I think might use the `icc` support in `numpy.distutils` and have thoughts...

Yeah, makes sense. Maybe if there was some good way for us to detect architecture, this could be done under the hood. Alternatively, we could add another environment variable.

Also, I think I may be confused here, but what is the problem with not enabling SSE2/SSE3 or AVX/AVX2 by default and simply letting the user select these? I think I agree with @juliantaylor that this should not be the default.

generic from source builds should always be default, you can make hardware availability arguments for some stuff like SSE3 (e.g. there are no SSE2 only intel macs) but in practice it doesn't matter, there is very little code in numpy that profits from anything better than SSE2.
hardware specific builds can be done by the user using the`OPT` environment variable.

@njsmith Yes, it generates code that only runs on CPUs that have all the codesets that the CPU used for compilation. However, I would stay away from automatic CPU dispatching, because it detects AMD cpus as SSE2, whatever they capable of.  

@jakirkham -xHOST automatically detects the architecture. Concerning the AVX issue, I just said, that it would restrict teh use of AVX on capable CPUs, so -xHOST is a better solution than -xSSE4.2

@hajgato 
-xHOST is not good variant: this option tells the compiler to generate instructions for the highest instruction set available on the compilation host processor. And you can use this option only in case, if you will use numpy/scipy only on your host or on the machine with the same CPU.
I think, that better solution is to use '/QaxSSE4.2' options on Windows and '-axSSE4.2' on Linux. In this case compiler should be generated code with SSE4.2 instructions for CPU with SSE4.2 support and generic code for another CPUs.
If you want to get specific parameters from environment, you can use this code:

``` python
import os
self.cc_exe = 'icc -m64 -fPIC -fp-model strict -O3 -fomit-frame-pointer -openmp ' + os.environ.get('ARCH_FLAGS', '-axSSE4.2')
```

@dzagorny Except `-axSSE4.2` and `/QaxSSE4.2` will not work on AMD CPUs, since the Intel automatic  CPU dispatching will use the slowest code on all AMD (Non-Intel) cpus.

@hajgato I'm not sure, we discuss about common parameters or about parameters, specific for your case?
About common: change from 'xSSE4.2' to os.environ.get('ARCH_FLAGS', '-axSSE4.2'). Code will be generated for SSE4.2 and another CPUs. If it is not good in several cases - user can specify required parameters in environment.
About specific cases: if you want to build and use numpy in the same machine - you can use -xHOST option.

@dzagorny 
I am already compiling numpy with `-xHOST` option (I do not remember which version of `numpy` I have started, but more than 2 years). 
Generally speaking: codes compiled with `-axSSE4.2` option will not run SSE4.2 code on CPUs that capable to run SSE4.2 and are Non-Intel CPUs. Whether is it a bad solution or not, it is not my duty to decide. I just told a remark about the behavior of `-ax` switch of Intel compilers.
All in all, the currently hard coded `-xSSE4.2` the worst solution at all, because then `numpy` does not run on non SSE4.2 capable CPU, whatever solution will be chosen that will ensure that `numpy` will not be restricted to "non-generic" cpu is fine for me. 
Additionally, why to restrict the code to SSE4.2? Why not compile to the newest available code, and run the older (slower) code on those CPUs that are not able to run it (or Non-Intel CPUs)? Why not to use for example `-axAVX`, `-axCORE-AVX2`, or even `-axCORE-AVX512`?

Any resolution as to what should be done here? I'm asking for the 1.11.1 release.

@charris according to me the best solution would be simply leave out any CPU specific optimization, and do it for general architecture by default.

Could someone suggest the proper flag to use, or better yet, make a PR?

Last call for 1.11.1. @hajgato @dzagorny.

@charris 
My suggestion is (No enforced architecture):

```
--- ./numpy/distutils/intelccompiler.py.orig    2016-01-06 22:07:54.000000000 +0100
+++ ./numpy/distutils/intelccompiler.py 2016-02-19 15:07:00.438235792 +0100
@@ -54,7 +54,7 @@
     def __init__(self, verbose=0, dry_run=0, force=0):
         UnixCCompiler.__init__(self, verbose, dry_run, force)
         self.cc_exe = ('icc -m64 -fPIC -fp-model strict -O3 '
-                       '-fomit-frame-pointer -openmp -xSSE4.2')
+                       '-fomit-frame-pointer -openmp')
         compiler = self.cc_exe
         if platform.system() == 'Darwin':
             shared_flag = '-Wl,-undefined,dynamic_lookup'
--- ./numpy/distutils/fcompiler/intel.py.orig   2016-01-07 03:16:25.000000000 +0100
+++ ./numpy/distutils/fcompiler/intel.py        2016-02-19 15:07:16.424220745 +0100
@@ -123,7 +123,7 @@
         return ['-openmp -fp-model strict']

     def get_flags_arch(self):
-        return ['-xSSE4.2']
+        return ['']

 # Is there no difference in the version string between the above compilers
 # and the Visual compilers?
```


Resolving patch:
diff --git a/numpy/distutils/fcompiler/intel.py b/numpy/distutils/fcompiler/intel.py
--- a/numpy/distutils/fcompiler/intel.py
+++ b/numpy/distutils/fcompiler/intel.py
@@ -123,7 +123,7 @@ def get_flags_opt(self):  # Scipy test failures with -O2
         return ['-openmp -fp-model strict -O1']
 
     def get_flags_arch(self):
-        return ['-xSSE4.2']
+        return ['']
 
 # Is there no difference in the version string between the above compilers
 # and the Visual compilers?
@@ -205,7 +205,7 @@ class IntelEM64VisualFCompiler(IntelVisualFCompiler):
     version_match = simple_version_match(start='Intel\(R\).*?64,')
 
     def get_flags_arch(self):
-        return ['/QaxSSE4.2']
+        return ['']
 
 
 if __name__ == '__main__':
diff --git a/numpy/distutils/intelccompiler.py b/numpy/distutils/intelccompiler.py
--- a/numpy/distutils/intelccompiler.py
+++ b/numpy/distutils/intelccompiler.py
@@ -54,7 +54,7 @@ class IntelEM64TCCompiler(UnixCCompiler):
     def __init__(self, verbose=0, dry_run=0, force=0):
         UnixCCompiler.__init__(self, verbose, dry_run, force)
         self.cc_exe = ('icc -m64 -fPIC -fp-model strict -O3 '
-                       '-fomit-frame-pointer -openmp -xSSE4.2')
+                       '-fomit-frame-pointer -openmp')
         compiler = self.cc_exe
         if platform.system() == 'Darwin':
             shared_flag = '-Wl,-undefined,dynamic_lookup'
@@ -88,7 +88,7 @@ def initialize(self, plat_name=None):
             self.lib = self.find_exe('xilib')
             self.linker = self.find_exe('xilink')
             self.compile_options = ['/nologo', '/O3', '/MD', '/W3',
-                                    '/Qstd=c99', '/QaxSSE4.2']
+                                    '/Qstd=c99']
             self.compile_options_debug = ['/nologo', '/Od', '/MDd', '/W3',
                                           '/Qstd=c99', '/Z7', '/D_DEBUG']
 


Test patch: