Repository: numpy/numpy

Issue:
Fortran Array problem in numpy 1.10
Hi,

I think I got a problem related to the numpy function np.asfortranarray. 
I am using numpy 1.10 with a pacakge named SPAMS(http://spams-devel.gforge.inria.fr/), it keep complaining that the input array is not a fortran array, but I used np.asfortranarray to convert it indeed. Then I tried to downgrade to numpy 1.9.3, it worked quite well, so I think there is a bug introduced since 1.10.

To reproduce the problem:

```
wget https://gforge.inria.fr/frs/download.php/latestfile/4532/spams-python-v2.5-svn2014-07-04.tar.gz
tar -xvf spams-python-v2.5-svn2014-07-04.tar.gz
cd spams-python
sudo python setup.py install
```

To run the test, you need to "cd spams-python" and them run the following code:

```
from test_dictLearn import *
test_trainDL()
```

And if you are using numpy 1.10 then it will complain an error immediately, if you use numpy 1.9.3 then it will run the test.

Thanks in advance to look into this.

Best,
Will



Discussion:
I probably have the same problem with different symptoms:
I wrote a code with extensive usage of structured arrays. When upgrading from numpy 1.9.3 to 1.10.0 the time it takes to run the code increases from 20 to 800 seconds.

'_get_all_field_offsets' and '_check_field_overlap' with the methods 'range' and 'difference' taking up most of the time after the update. Seems like some security function when manipulating structured arrays or maybe lists. Would be nice if you could check the reason. I am not so familiar with the details.

I upload 2 cProfile logs of my code. One for 1.9 and one for 1.10 numpy version with no other changes. You can find them here:
https://www.dropbox.com/sh/8regcosqmn616i9/AAC_OzJumGQsoban1V2khbsea?dl=0

@SIKz17, happily your problem is known and already fixed (see #6467)! The slowdown should disappear when you upgrade to 1.10.2 which should be released very soon.

@oeway can you slim down the test to the simplest code that reproduces the error?

I ran your scripts, but I get the following error:

```
RuntimeError: matrix arg 7 must be a 2d double Fortran Array
```

which does not look like a problem in numpy at first glance.

OK, I think I found the reproducible code:

```
>>> a = np.asfortranarray(np.empty((2, 0), dtype=np.float64))
>>> np.isfortran(a)
False
>>> a.flags
 C_CONTIGUOUS : True
 F_CONTIGUOUS : True
 OWNDATA : True
 WRITEABLE : True
 ALIGNED : True
 UPDATEIFCOPY : False
```

Sounds like a relaxed strides issue then. And likely a bug in isfortran?
(On my phone so can't check the code, but there is also a bug in the
isfortran docstring in that it explicitly documents that you can't have an
array that is both C and Fortran ordered, which is no longer true.)
On Oct 30, 2015 8:56 AM, "ahaldane" notifications@github.com wrote:

> OK, I think I found the reproducible code:
> 
> > > > a = np.asfortranarray(np.empty((2, 0), dtype=np.float64))
> > > > np.isfortran(a)
> > > > False
> > > > a.flags
> > > >  C_CONTIGUOUS : True
> > > >  F_CONTIGUOUS : True
> > > >  OWNDATA : True
> > > >  WRITEABLE : True
> > > >  ALIGNED : True
> > > >  UPDATEIFCOPY : False
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/numpy/numpy/issues/6590#issuecomment-152564545.

Looking more closely, I think this may confusing (but documented) behavior of isfortran, rather than a numpy bug. At the bottom of the [isfortran docstring](http://docs.scipy.org/doc/numpy/reference/generated/numpy.isfortran.html) it says that "C-ordered arrays evaluate as False even if they are also FORTRAN-ordered."

I vaguely remember seeing code in f2py that requires fortran ordered arrays, but it checks this by looking at the `F_CONTIGUOUS` flag rather than using the `isfortran` function for this reason.

So it is possible that SPAMS should replace its calls to `isfortran` by checks of the `F_CONTIGUOUS` flag. But I am not very familiar with the fortran interface so could be wrong. Does anyone more familiar know?

Oh @njsmith I didn't see you comment. As far as I can tell the current behavior in 1.10 matches the docstring, eg

```
>>> np.isfortran(np.arange(10))
False
>>> np.__version__
'1.10.1'
```

The function calls `a.flags.fnc`, which I suspect is very old. I suspect `fnc` means `Fortran not C`, and I'm not clear on why anyone would want to use it.

Is it even possible to have a 1D, contiguous array for which `flags.fnc`
evaluates to true?

On Fri, Oct 30, 2015 at 12:18 PM, Charles Harris notifications@github.com
wrote:

> The function calls a.flags.fnc, which I suspect is very old. I suspect fnc
> means Fortran not C, and I'm not clear on why anyone would want to use it.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/numpy/numpy/issues/6590#issuecomment-152571751.

@ewmoore  No, not any more. The documentation probably documents what the functions _does_, not what it _should do_. If nothing else, the name is horribly misleading. It would be interesting to see what the Numeric documentation says.

Which is to say, it could be a bug dating back to 2005.

Documentation in 1.0.1

```
Returns True if 'a' is arranged in Fortran-order in memory with a.ndim > 1
```

Which I suspect is not quite right either.

For what it's worth, defining `isfortran` as

```
def isfortran(a):
    return a.flags['F_CONTIGUOUS']
```

does not cause any unit tests to fail in numpy or scipy in python2.

To me this all sounds like collateral damage from the old "strict strides" handling: if you artificially make a rule that the same array can't be both C and Fortran ordered at the same time, then it makes sense to have a `fnc` operation, and to say things like "only arrays with >2d can be Fortran ordered" (because before the advent of the flag system, all contiguous 1d arrays were C contiguous, therefore they couldn't also be Fortran contiguous). None of this really makes any sense, but it's at least internally coherent if you squint at it a bit sideways.

In the new relaxed strides world it doesn't make any sense at all, though. I think we just have to fix isfortran to mean, well, isfortran, or else deprecate and remove it entirely. Fortran-not-C is just not a useful thing for a function with that name to be checking at this point.

I'm a little nervous about changing isfortran in a point release though -- hard to know how much stuff might be out there with weird dependencies in the current behavior :-/

@teoliphant Maybe one of the great old ones knows the history.

@njsmith Yeah, no changes for 1.10.

The `fnc` method belongs to the `flags` object and cannot be made backwards compatible. I suggest we deprecate it for removal. The most compatible fix for `isfortran` is

```
def isfortran(a):
    return a.flags['F_CONTIGUOUS'] and a.ndims > 1
```

I suggest we make that change for 1.10. We might also want to raise a `FutureWarning` and drop the requirement on `ndims` at some point.

On Fri, Oct 30, 2015 at 4:19 PM, argriffing notifications@github.com
wrote:

> Return a.flags.f_contiguous and a.ndim > 1, which would be backward
> compatible.
> 
> This is not really backwards compatible, right?
> 
> > > > a = np.zeros((2, 1))>>> a.flags['F_CONTIGUOUS'] and a.ndim > 1True>>> np.isfortran(a)False>>> a.flags.fncFalse
> 
>  Hmm, you are right.

Fortran ordering was a new thing in 2005.   C-order was still the default for most things internally.  From what I recall, the fnc flag was intended to be used for situations when you wanted to force a Fortran order in a new construct based on another array (i.e. you pass the arr.flags.fnc into an argument that created a new array in either Fortran or C order based on that flag).   I'm pretty sure that was the main reason it was created.    

There was definitely an assumption that a "fortran array" statisfied a.ndim > 1.    Because while 1-d C-contiguous was also F-contiguous, originally C-contiguous was still preferred and so the array was considered C-contiguous.     Fortran arrays were asymmetrically supported. 

I don't really understand the motivating use-cases for relaxed stride checking, but there is definitely logical work to do to reintroduce the same "isfortran" concept that existed without relaxes stride checking.  I suppose you could add-back the "strictness"  of the previous use of the Fortran Flag --- for backward compatibility that would make sense. 

I don't know what other motivating use-cases are driving isfortran right now and whether or not isfortran was involved in the discussion around introducing relaxed stride checking.     

The use for which isfortran was created was to determine whether or not to transpose the memory of an array with ndim > 1 before passing it into a low-level routine that expected Fortran arrays.  

I would suggest that @charris fix is the right one and that fnc should definitely be deprecated (as it is not the right concept now).   For the cases that "break" backward compatibility --- i.e where np.flags.fnc returns False but (a.flags['F_CONTIGUOUS'] and a.ndim > 1) returns True --- the difference is a bug in the previous code. 

But, of course semantic changes like relaxed stride checking will create these kinds of difficult decisions where you try to judge exactly whose code to break.   As far as things go, as long as it's documented, this should be a generally easy thing for people to fix downstream if @charris proposed change to isfortran is merged.

@ahaldane Ah, and in 1.9

```
In [1]: a = np.ones((2, 0), dtype=np.float64)

In [2]: a.flags
Out[2]: 
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [3]: asfortranarray(a).flags
Out[3]: 
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [4]: np.__version__
Out[4]: '1.9.2'

```

I would suggest that 1.9.0 is being inconsistent here.

EDIT: and it is making a copy. Also

```
In [9]: a = np.ones((2, 0), dtype=np.float64, order='F')

In [10]: a.flags
Out[10]: 
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False
```

By the way I think it might be important to note that this bug occurs in a SWIG interface, and actually does not directly involve `isfortran`, but rather `PyArray_ISFORTRAN` (which does the same thing in C). 

To add an extra layer of complexity, the code actually calls the `array_is_fortran` swig interface function in `tools/swig/numpy.i`, which is documented [here](http://docs.scipy.org/doc/numpy/reference/swig.interface-file.html#macros) as "Evaluates as true if a is FORTRAN ordered.", which (as we have been discussing) is not really true.

Here is the actual code that fails on the array we are looking at:

``` c
/*@SWIG:py_typemaps.i,86,map_matrix@*/
array7 = obj_to_array_no_conversion(obj3, NPY_DOUBLE);
/* !!!!!
    WARNING! bug (?) : the variable name choosen above must not appear
    in the string, otherwise swig will not correctly generate
    final variable names (above name + number)
    */
/* we cannot use require_fortran, because it convert a numpy C array7 to a numpy 
    fortran array7 by just modifying the strides */
if (!array7 || !require_dimensions(array7,2) || !array_is_fortran(array7) || !require_native(array7)) {
  SWIG_Python_SetErrorMsg(PyExc_RuntimeError,"matrix arg 7 must be a 2d double Fortran Array"); SWIG_fail;
}
arg7 = new Matrix<double> ((double *)array_data(array7),(int)array_size(array7,0),(int)array_size(array7,1));

/*@SWIG@*/
```

~~(I do not think we have access to the original swig file, only the generated code)~~. Edit: The original code is [here](https://github.com/gbkedar/spams_toolbox_with_cmake_cpp_wrap/blob/master/swig/python/py_typemaps.i#L86)

Judging from the comment the function `require_fortran` is also confusing.

It is possible that modifying `array_is_fortran` in `numpy.i` is a way out of back-compatibility problems.

By the way the documentation for `PyArray_ISFORTRAN` is much more to the point:

"Evaluates true if arr is Fortran-style contiguous and not C-style contiguous. `PyArray_IS_F_CONTIGUOUS` is the correct way to test for Fortran-style contiguity."

So it does sound like `array_is_fortran` is not implemented correctly.

The misleading behavior of `np.isfortran` is related, but fixing `isfortran` would not fix the particular bug in this issue.

@ahaldane Thanks for tracking that down. From the lawyers point of view "Evaluates true if arr is Fortran-style contiguous and not C-style contiguousv" is still valid, we are just being more inclusive as to which arrays are c and f contiguous. At this point I think 1) fixing `array_is_fortran` and 2) changing `isfortran` to simply check fortran contiguity will solve most compatibility problems.


Resolving patch:
diff --git a/numpy/core/numeric.py b/numpy/core/numeric.py
--- a/numpy/core/numeric.py
+++ b/numpy/core/numeric.py
@@ -696,8 +696,12 @@ def require(a, dtype=None, requirements=None):
 
 def isfortran(a):
     """
-    Returns True if array is arranged in Fortran-order in memory
-    and not C-order.
+    Returns True if the array is Fortran contiguous but *not* C contiguous.
+
+    This function is obsolete and, because of changes due to relaxed stride
+    checking, its return value for the same array may differ for versions
+    of Numpy >= 1.10 and previous versions. If you only want to check if an
+    array is Fortran contiguous use ``a.flags.f_contiguous`` instead.
 
     Parameters
     ----------


Test patch: