Repository: Theano/Theano

Issue:
cutils fails when cutils_ext already present elsewhere in sys.path
EDIT: Now it work with the c|py linker, but not with the cvm linker.

Theano's cutils moudle expects to find cutils_ext inside config.compiledir.

In cutils.py, cutils_ext.cutils_ext will be loaded.  On an import error, this extension will be built and eventually cmodule.dlimport will re-import cutils_ext.cutils_ext.  dlimport will inject config.compiledir into the front of sys,path, so the cutils_ext should always be loaded from config.compiledir.

Normally, this will then allow cutils_ext.cutils_ext to be loaded.  The problem is if the cutils_ext package was found elsewhere on sys.path, but not the cutils_ext module (extension) within that package.  cutils.py lacks the sys.path injection of dlimport.  The consequence is that the import of cutils_ext.cutils_ext in dlimport fails because the reference to the outer cutils_ext (the package) will be the incorrect path!

Needless to say, when a user encounters this bug, it is a pain to solve.

The solution is to ensure that the cutils.py try statement starts with:

sys.path.insert(0,theano.config.compiledir)
with the finally doing:
del sys.path[0]



Discussion:
A pain to solve indeed! Thanks for taking the time to track it down,
and report it. It's a long weekend in Canada, but hopefully Fred or
Pascal can patch this into the trunk next week.
- James

On Fri, Jun 29, 2012 at 10:11 PM, usaar33
reply@reply.github.com
wrote:

> Theano's cutils moudle expects to find cutils_ext inside config.compiledir.
> 
> In cutils.py, cutils_ext.cutils_ext will be loaded.  On an import error, this extension will be built and eventually cmodule.dlimport will re-import cutils_ext.cutils_ext.  dlimport will inject config.compiledir into the front of sys,path, so the cutils_ext should always be loaded from config.compiledir.
> 
> Normally, this will then allow cutils_ext.cutils_ext to be loaded.  The problem is if the cutils_ext package was found elsewhere on sys.path, but not the cutils_ext module (extension) within that package.  cutils.py lacks the sys.path injection of dlimport.  The consequence is that the import of cutils_ext.cutils_ext in dlimport fails because the reference to the outer cutils_ext (the package) will be the incorrect path!
> 
> Needless to say, when a user encounters this bug, it is a pain to solve.
> 
> The solution is to ensure that the cutils.py try statement starts with:
> 
> sys.path.insert(0,theano.config.compiledir)
> with the finally doing:
> del sys.path[0]
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/Theano/Theano/issues/728

Thanks for the report. In the trunk version of cutils, we append the path to sys.path if it is not present. I'll change that to an insert(0, path) as you suggest and delete it when we finish with it.

Also, what other project used the same name for as this module?

@jaberg, can you review the PR for this: gh-739?

thanks

The bug wasn't raised during a project; it had to do with how PiCloud (http://www.picloud.com) was handling things.  PiCloud automatically transports python modules it finds to the server.  It didn't transport the .so of course, but a package (in the sense of having an **init**.py) called cutils_ext was created in its "automatic packages" directory.  When a user then would use theano on PiCloud, the incorrect package would load.

I don't understand everything you said, is the cutils_ext was created by PiCloud itself?

Anyway, can you look at gh-739 and tell us if you that this fix will solve the problem?

thanks

The patch appears correct.

I didn't provide enough context in my explaination. Here is how PiCloud transports modules: http://docs.picloud.com/tech_overview.html#language-integration

So yes, a cutils_ext package (without the cutils_ext.so extension module) is created by PiCloud (as part of the automatic module transport from the client machine to its servers).

thank for the information.

I was experimenting with picloud this afternoon; I get the error described in this issue (importerror on cutils_ext)

This is my test code, if anyone cares to reproduce:

``` python
import theano
import theano.tensor as tt
import cloud
x = tt.scalar()
f = lambda y : theano.function([x], x)(y)
jid = cloud.call(f, 1)
cloud.result(jid)
```

Witch version of Theano do you use? 0.6rc1?

Yes, 0.6rc1.

Sadly, I don't have the time to look at this shortly. Do you want to try to identify the problem? Can you test with the version when we closed this ticket, it was supposed to work at that time. If it work, can you bisect to find the commit that broke this?

thanks

As far as I can tell, this actually didn't work at the time (may not have been fully tested). However, I worked with PiCloud to build a wrapper function that properly inserted cutils_ext. The wrapper function worked -- meaning the code runs and returns the correct result -- at the time this ticket was closed, but no longer works (something about a global 'CVM' not being defined that must be related to the CVM linker). 

This isn't a priority for me right now, but when I have some time I will explore it and see if I can find why it doesn't work.

If it is about the cvm, you can try to disable it with this theano flag:

linker=c|py

The CVM ask the compilation of a new file in: COMPILEDIR/lazylinker_ext/. I tell this just in case it help to understand the problem.


Resolving patch:
diff --git a/theano/gof/cutils.py b/theano/gof/cutils.py
--- a/theano/gof/cutils.py
+++ b/theano/gof/cutils.py
@@ -67,7 +67,8 @@ def compile_cutils():
     location = os.path.join(config.compiledir, 'cutils_ext')
     if not os.path.exists(location):
         os.mkdir(location)
-    file(os.path.join(location, "__init__.py"), 'w').close()
+    if not os.path.exists(os.path.join(location, '__init__.py')):
+        file(os.path.join(location, '__init__.py'), 'w').close()
 
     try:
         from cutils_ext.cutils_ext import *
@@ -80,7 +81,7 @@ def compile_cutils():
     # compile the cutils_ext module simultaneously.
         try:
             try:
-                # We must retry to import it as some other processs could
+                # We must retry to import it as some other process could
                 # have been compiling it between the first failed import
                 # and when we receive the lock
                 from cutils_ext.cutils_ext import *
diff --git a/theano/gof/lazylinker_c.py b/theano/gof/lazylinker_c.py
--- a/theano/gof/lazylinker_c.py
+++ b/theano/gof/lazylinker_c.py
@@ -22,6 +22,19 @@ def try_reload():
     del sys.path[0]
 
 try:
+    # See gh issue #728 for why these lines are here. Summary: compiledir must
+    # be at the beginning of the path to avoid conflicts with any other
+    # lazylinker_ext modules that might exist (this step handled in try_import
+    # and try_reload). An __init__.py file must be created for the same reason.
+    # Note that these lines may seem redundant (they are repeated in
+    # compile_str()) but if another lazylinker_ext does exist then it will be
+    # imported and compile_str won't get called at all.
+    location = os.path.join(config.compiledir, 'lazylinker_ext')
+    if not os.path.exists(location):
+        os.mkdir(location)
+    if not os.path.exists(os.path.join(location, '__init__.py')):
+        file(os.path.join(location, '__init__.py'), 'w').close()
+
     _need_reload = False
     if force_compile:
         raise ImportError()


Test patch: