Repository: Theano/Theano

Issue:
`.../c_code/dimshuffle.c` missing from setup.py?
A Lasagne user just reported that
```bash
pip install --upgrade --no-deps https://github.com/Theano/Theano/archive/master.zip
```
lead to the following error (I assume after installation, when trying to import it):
```
FileNotFoundError: [Errno 2] No such file or directory: '/Users/XXXX/anaconda/lib/python3.5/site-packages/theano/tensor/c_code/dimshuffle.c
```
The file was added in #6174 and is included in the `master.zip` snapshot, but maybe it's missing in `setup.py` and thus not copied over? You wouldn't notice when doing a development/editable install. The user solved it by downloading `tensor/c_code/dimshuffle.c` and `gpuarray/c_code/dimshuffle.c` manually from github.

Disclaimer: I haven't verified this behavior myself.
Add "c_code" folders to Theano installations
To do that, I converted current `c_code` folders to python modules (I don't find other solution, currently). This should fix #6222 . I had forgotten that @abergeron warned about it in #6047.

@nouiz @lamblin 


Discussion:
I confirm the problem.
> To do that, I converted current `c_code` folders to python modules (I don't find other solution, currently). 

But that will introduce `theano.gpuarray.c_code` as an empty module. Can't you just adapt the `find_packages` function in `setup.py` (https://github.com/Theano/Theano/blob/master/setup.py#L62) so it not only looks for `__init__.py`, but also for `c_code`?
Thank you for updating the PR, but it seems this doesn't work, I'm just testing it. I've found another way. Will send a PR.
@f0k  It works for me with new changes that you suggested (ie. using `find_packages()`):

```
pip install --upgrade --no-deps https://github.com/notoraptor/Theano/archive/fix-setup-for-c-code-folders.zip
```
Funny. Yes, I can confirm it works. I tried it locally before and it didn't work, but now it does, not sure what was wrong. I assumed distutils still expects a `__init__.py`, as mentioned in the documentation: https://docs.python.org/2/distutils/setupscript.html#listing-whole-packages. But apparently it's not so strict, or it's different in setuptools.

In any case, the officially supported way is to use `package_data` instead: https://docs.python.org/2/distutils/setupscript.html#installing-package-data.
That would be:
```diff
diff --git a/setup.py b/setup.py
index bf26983..26b65ed 100755
--- a/setup.py
+++ b/setup.py
@@ -167,12 +167,12 @@ def do_setup():
               'test': ['nose>=1.3.0', 'nose-parameterized>=0.5.0', 'flake8<3'],
               'doc': ['Sphinx>=0.5.1', 'pygments']
           },
-          package_data={
-              '': ['*.txt', '*.rst', '*.cu', '*.cuh', '*.c', '*.sh', '*.pkl',
-                   '*.h', '*.cpp', 'ChangeLog'],
-              'theano.misc': ['*.sh'],
-              'theano.d3viz' : ['html/*','css/*','js/*']
-          },
+          package_data=dict([
+              ('', ['*.txt', '*.rst', '*.cu', '*.cuh', '*.c', '*.sh', '*.pkl',
+                    '*.h', '*.cpp', 'ChangeLog']),
+              ('theano.misc', ['*.sh']),
+              ('theano.d3viz', ['html/*','css/*','js/*']),
+          ] + [(package, ['c_code/*.c']) for package in find_packages()]),
           entry_points={
               'console_scripts': ['theano-cache = bin.theano_cache:main',
                                   'theano-nose = bin.theano_nose:main']
```
I can send a PR with this if you'd prefer that solution for some reason.
@f0k Does your patch work ? If so, you can send your PR and I will close that current one.
It works, yes, but I'm not sure if it's nicer than yours :) It blindly adds `c_code/*.c` for all theano packages, no matter if it exists or not, and the implementation is a little less obvious. I'm leaning towards keeping your solution until somebody complains.
Ok!

@abergeron @nouiz @lamblin what do you think ?
I think using package_data is a better solution in theory, but I've had a bunch of problem with libgpuarray where package_data would end up in a strange folder somewhere else in the filesystem.

I'm just worried that some version of setuptools and/or distutils is going to check for `__init__.py` and complain or refuse to work when it's not there.
@abergeron So, which solution should we use ? Convert any `c_code` folder to a module by adding an `__init__.py` file ?
I think I found another way, wait a minute...
Yes, it works. PR underway.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -169,7 +169,7 @@ def do_setup():
           },
           package_data={
               '': ['*.txt', '*.rst', '*.cu', '*.cuh', '*.c', '*.sh', '*.pkl',
-                   '*.h', '*.cpp', 'ChangeLog'],
+                   '*.h', '*.cpp', 'ChangeLog', 'c_code/*'],
               'theano.misc': ['*.sh'],
               'theano.d3viz' : ['html/*','css/*','js/*']
           },


Test patch: