Repository: Theano/Theano

Issue:
No ldflags and no fallback for [sd]gemv_ on Windows
I have a compilation error on Theano when trying to test tensor corr operations on Windows 10 64 bits:

```

(python2) C:\Users\notoraptor\mila\dev\git\theano>nosetests -xvs theano\tensor\nnet\tests\test_corr.py
test_fwd (theano.tensor.nnet.tests.test_corr.Grouped_conv_noOptim) ...
You can find the C code in this temporary file: c:\users\notora~1\appdata\local\temp\theano_compilation_error_fkklco
ERROR

======================================================================
ERROR: test_fwd (theano.tensor.nnet.tests.test_corr.Grouped_conv_noOptim)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\tensor\nnet\tests\test_abstract_conv.py", line 1491, in test_fwd
    mode=self.ref_mode)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\compile\function.py", line 317, in function
    output_keys=output_keys)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\compile\pfunc.py", line 486, in pfunc
    output_keys=output_keys)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\compile\function_module.py", line 1842, in orig_function
    fn = m.create(defaults)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\compile\function_module.py", line 1716, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\gof\link.py", line 699, in make_thunk
    storage_map=storage_map)[:3]
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\gof\vm.py", line 1084, in make_all
    impl=impl))
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\gof\op.py", line 955, in make_thunk
    no_recycling)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\gof\op.py", line 858, in make_c_thunk
    output_storage=node_output_storage)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\gof\cc.py", line 1217, in make_thunk
    keep_lock=keep_lock)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\gof\cc.py", line 1157, in __compile__
    keep_lock=keep_lock)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\gof\cc.py", line 1620, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\gof\cmodule.py", line 1174, in module_from_key
    module = lnk.compile_cmodule(location)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\gof\cc.py", line 1523, in compile_cmodule
    preargs=preargs)
  File "C:\Users\notoraptor\mila\dev\git\theano\theano\gof\cmodule.py", line 2353, in compile_str
    (status, compile_stderr.replace('\n', '. ')))
Exception: ('The following error happened while compiling the node', CorrMM{valid, (1, 1), (1, 1), 1 False}(img, Subtensor{::, ::, ::int64, ::int64}.0), '\n', "Compilation failed (return status=1): C:\\Users\\NOTORA~1\\AppData\\Local\\Temp\\ccxCBz0C.o: In function `corrMM(tagPyArrayObject*, tagPyArrayObject*, tagPyArrayObject*, int, int, int, int, int, int, int, int, int)':\r. C:/Users/notoraptor/AppData/Local/Theano/compiledir_Windows-10-10.0.15063-Intel64_Family_6_Model_158_Stepping_9_GenuineIntel-2.7.13-64/tmpzoeqsr/mod.cpp:1199: undefined reference to `dgemv_'\r. collect2.exe: error: ld returned 1 exit status\r. ", '[CorrMM{valid, (1, 1), (1, 1), 1 False}(img, <TensorType(float64, 4D)>)]')

----------------------------------------------------------------------
Ran 1 test in 1.956s

FAILED (errors=1)
```

Error message says:
```
undefined reference to `dgemv_'\r. collect2.exe: error: ld returned 1 exit status
```

When checking theano.config.blas.ldflags, this flag appeared to be empty:
```

(python2) C:\Users\notoraptor\mila\dev\git\theano>python
Python 2.7.13 |Continuum Analytics, Inc.| (default, May 11 2017, 13:17:26) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import theano
>>> theano.config.blas.ldflags
''
```

Even if conda packages mkl, mkl-service and m2w64-openblas were installed.
```

(python2) C:\Users\notoraptor\mila\dev\git\theano>conda list
# packages in environment at C:\Users\notoraptor\data\programs\Anaconda3\envs\python2:
#
m2w64-openblas            0.2.19                        1

mkl                       2017.0.3                      0
mkl-service               1.1.2                    py27_3
```

So, it seems Theano cannot set ldflags on Windows even when BLAS libraries are installed. Other useful remarks:
- MKL packages seems to contains only DLL files, no `.lib` files. So, it seems quite impossible to link against MKL files.
- Theano currently provides a fallback for `[sd]gemm` when ldflags is empty, but not for `[sd]gemv_` as it was not used. But the new feature about unshared convolutions needs `[sd]gemv_` functions. So, maybe should we implement a fallback for these functions too.

@abergeron @nouiz @lamblin 


Discussion:
This is because MKL linking fails on windows.

I think we should add sgemv and dgemv to the fallback code, but I don't think there is anything we can do for mkl linking.

Resolving patch:
diff --git a/theano/tensor/blas_c.py b/theano/tensor/blas_c.py
--- a/theano/tensor/blas_c.py
+++ b/theano/tensor/blas_c.py
@@ -444,7 +444,7 @@ def gemv_c_code(y, A, x, z, alpha, beta, fail,
         dtype_%(x)s* x_data = (dtype_%(x)s*) PyArray_DATA(%(x)s);
         dtype_%(z)s* z_data = (dtype_%(z)s*) PyArray_DATA(%(z)s);
         // gemv expects pointers to the beginning of memory arrays,
-        // but numpy provides provides a pointer to the first element,
+        // but numpy provides a pointer to the first element,
         // so when the stride is negative, we need to get the last one.
         if (Sx < 0)
             x_data += (NA1 - 1) * Sx;
diff --git a/theano/tensor/blas_headers.py b/theano/tensor/blas_headers.py
--- a/theano/tensor/blas_headers.py
+++ b/theano/tensor/blas_headers.py
@@ -731,29 +731,26 @@ def cblas_header_text():
 def blas_header_text():
     """C header for the fortran blas interface"""
 
-    gemm_code = ""
-    const = "const"
+    blas_code = ""
     if not config.blas.ldflags:
         # Include the Numpy version implementation of [sd]gemm_.
         current_filedir = dirname(__file__)
-        gemm_common_filepath = os.path.join(current_filedir, 'c_code', 'alt_gemm_common.c')
-        gemm_template_filepath = os.path.join(current_filedir, 'c_code', 'alt_gemm_template.c')
+        blas_common_filepath = os.path.join(current_filedir, 'c_code', 'alt_blas_common.h')
+        blas_template_filepath = os.path.join(current_filedir, 'c_code', 'alt_blas_template.c')
         common_code = ""
-        sgemm_code = ""
-        dgemm_code = ""
-        with open(gemm_common_filepath) as code:
+        sblas_code = ""
+        dblas_code = ""
+        with open(blas_common_filepath) as code:
             common_code = code.read()
-        with open(gemm_template_filepath) as code:
+        with open(blas_template_filepath) as code:
             template_code = code.read()
-            sgemm_code = template_code % {"float_type": "float", "float_size": 4, "npy_float": "NPY_FLOAT32", "name": "sgemm_"}
-            dgemm_code = template_code % {"float_type": "double", "float_size": 8, "npy_float": "NPY_FLOAT64", "name": "dgemm_"}
-        if not common_code or not sgemm_code:
-            raise IOError("Unable to load NumPy implementation of gemm code from C source files.")
-        else:
-            const = ""
-        gemm_code += common_code
-        gemm_code += sgemm_code
-        gemm_code += dgemm_code
+            sblas_code = template_code % {"float_type": "float", "float_size": 4, "npy_float": "NPY_FLOAT32", "precision": "s"}
+            dblas_code = template_code % {"float_type": "double", "float_size": 8, "npy_float": "NPY_FLOAT64", "precision": "d"}
+        if not common_code or not template_code:
+            raise IOError("Unable to load NumPy implementation of BLAS functions from C source files.")
+        blas_code += common_code
+        blas_code += sblas_code
+        blas_code += dblas_code
 
     header = """
     extern "C"
@@ -916,7 +913,7 @@ def blas_header_text():
 
     /* Single Precision */
 
-        void sgemm_(char*, char*, const int*, const int*, const int*, const float *, %(const)s float *, const int*, %(const)s float *, const int*, const float *, float *, const int*);
+        void sgemm_(char*, char*, const int*, const int*, const int*, const float *, const float *, const int*, const float *, const int*, const float *, float *, const int*);
         void ssymm_(char*, char*, const int*, const int*, const float *, const float *, const int*, const float *, const int*, const float *, float *, const int*);
         void ssyrk_(char*, char*, const int*, const int*, const float *, const float *, const int*, const float *, float *, const int*);
         void ssyr2k_(char*, char*, const int*, const int*, const float *, const float *, const int*, const float *, const int*, const float *, float *, const int*);
@@ -925,7 +922,7 @@ def blas_header_text():
 
     /* Double Precision */
 
-        void dgemm_(char*, char*, const int*, const int*, const int*, const double *, %(const)s double *, const int*, %(const)s double *, const int*, const double *, double *, const int*);
+        void dgemm_(char*, char*, const int*, const int*, const int*, const double *, const double *, const int*, const double *, const int*, const double *, double *, const int*);
         void dsymm_(char*, char*, const int*, const int*, const double *, const double *, const int*, const double *, const int*, const double *, double *, const int*);
         void dsyrk_(char*, char*, const int*, const int*, const double *, const double *, const int*, const double *, double *, const int*);
         void dsyr2k_(char*, char*, const int*, const int*, const double *, const double *, const int*, const double *, const int*, const double *, double *, const int*);
@@ -984,7 +981,11 @@ def blas_header_text():
                     }
                     """)
 
-    return (header % {'const': const}) + gemm_code
+    return header + blas_code
+
+
+if not config.blas.ldflags:
+    _logger.warning('Using NumPy C-API based implementation for BLAS functions.')
 
 
 def mkl_threads_text():
@@ -1032,7 +1033,7 @@ def openblas_threads_text():
 
 def blas_header_version():
     # Version for the base header
-    version = (2,)
+    version = (5,)
     if detect_macos_sdot_bug():
         if detect_macos_sdot_bug.fix_works:
             # Version with fix


Test patch:
diff --git a/theano/tensor/tests/test_blas_c.py b/theano/tensor/tests/test_blas_c.py
--- a/theano/tensor/tests/test_blas_c.py
+++ b/theano/tensor/tests/test_blas_c.py
@@ -316,5 +316,92 @@ def setUp(self):
         skip_if_blas_ldflags_empty()
 
 
+class TestCGemvNoFlags(object):
+    mode = mode_blas_opt
+    gemv = CGemv(inplace=False)
+    M = 4
+    N = 5
+    slice_step = 3
+
+    def setUp(self):
+        unittest_tools.seed_rng()
+
+    def get_function(self, dtype, transpose_A=False, slice_tensors=False):
+        alpha = theano.tensor.scalar(dtype=dtype)
+        beta = theano.tensor.scalar(dtype=dtype)
+        A = theano.tensor.matrix(dtype=dtype)
+        x = theano.tensor.vector(dtype=dtype)
+        y = theano.tensor.vector(dtype=dtype)
+        if transpose_A:
+            A_1 = A.T
+        else:
+            A_1 = A
+        if slice_tensors:
+            A_2 = A_1[::-self.slice_step]
+            x_2 = x[::-self.slice_step]
+            y_2 = y[::-self.slice_step]
+        else:
+            A_2 = A_1
+            x_2 = x
+            y_2 = y
+        return theano.function([alpha, A, x, beta, y], self.gemv(y_2, alpha, A_2, x_2, beta))
+
+    def get_data(self, dtype, alpha, beta, transpose_A=False, slice_tensors=False):
+        if slice_tensors:
+            if transpose_A:
+                A_shape = (self.N, self.M * self.slice_step)
+            else:
+                A_shape = (self.M * self.slice_step, self.N)
+            x_shape = (self.N * self.slice_step,)
+            y_shape = (self.M * self.slice_step,)
+        else:
+            if transpose_A:
+                A_shape = (self.N, self.M)
+            else:
+                A_shape = (self.M, self.N)
+            x_shape = (self.N,)
+            y_shape = (self.M,)
+        A = np.random.random(A_shape).astype(dtype)
+        x = np.random.random(x_shape).astype(dtype)
+        y = np.random.random(y_shape).astype(dtype)
+        return (alpha, A, x, beta, y)
+
+    def compute_ref(self, alpha, A, x, beta, y, transpose_A, slice_tensors):
+        if transpose_A:
+            A = A.T
+        if slice_tensors:
+            A = A[::-self.slice_step]
+            x = x[::-self.slice_step]
+            y = y[::-self.slice_step]
+        ref_val = alpha * np.dot(A, x)
+        if beta != 0:
+            ref_val += beta * y
+        return ref_val
+
+    @theano.change_flags({'blas.ldflags': ''})
+    def run_cgemv(self, dtype, ALPHA, BETA, transpose_A, slice_tensors):
+        f = self.get_function(dtype, transpose_A=transpose_A, slice_tensors=slice_tensors)
+        values = self.get_data(dtype, ALPHA, BETA, transpose_A=transpose_A, slice_tensors=slice_tensors)
+        assert any(isinstance(node.op, CGemv) for node in f.maker.fgraph.apply_nodes)
+        z_val = f(*values)
+        assert z_val.dtype == dtype
+        assert z_val.ndim == 1
+        assert z_val.shape[0] == self.M
+        ref_val = self.compute_ref(*(values + (transpose_A, slice_tensors)))
+        unittest_tools.assert_allclose(ref_val, z_val)
+
+    def test_cgemv(self):
+        for dtype in ('float32', 'float64'):
+            for alpha in (0, 1, -2):
+                for beta in (0, 1, -2):
+                    for transpose_A in (False, True):
+                        for slice_tensors in (False, True):
+                            yield (self.run_cgemv, dtype, alpha, beta, transpose_A, slice_tensors)
+
+
+class TestSdotNoFlags(TestCGemvNoFlags):
+    M = 1
+
+
 class TestBlasStridesC(TestBlasStrides):
     mode = mode_blas_opt