Repository: piskvorky/gensim

Issue:
import gensim fails since updating to Xcode 7.3 
I just updated my version of Xcode to 7.3. When I run `pip install --upgrade gensim` the process completed without any issues. However, when I try `import gensim` within the python shell the terminal barfs a bunch of C++ output with a block of execution errors that begins with: 

`Exception: Compilation failed (return status=1): clang: error: unsupported option '-b mi2'. clang: error: unsupported option '-b mi'. clang: error: unknown argument: '-target-feature'. clang: error: unknown argument: '-sse4a'. clang: error: unknown argument: '-target-feature'. clang: error: unknown argument: '-target-feature'. clang: error: unknown argument: '-target-feature'. clang: error: unknown argument: '-tbm'. clang: error: unknown argument: '-target-feature'. clang: error: unknown argument: '-target-feature'....`

I think this has something to do with where gensim is looking for its header files, but I'm somewhat at a loss. Any help debugging would be greatly appreciated. 



Discussion:
Updated to include a python trace of the error progression: 

---

Exception                                 Traceback (most recent call last)
<ipython-input-1-0539d76422c6> in <module>()
----> 1 import gensim

/Users/aaronpolhamus/Documents/repos/gensim/gensim/**init**.py in <module>()
      4 """
      5 
----> 6 from gensim import parsing, matutils, interfaces, corpora, models, similarities, summarization
      7 import logging
      8 

/Users/aaronpolhamus/Documents/repos/gensim/gensim/corpora/**init**.py in <module>()
     16 from .ucicorpus import UciCorpus
     17 from .malletcorpus import MalletCorpus
---> 18 from .sharded_corpus import ShardedCorpus

/Users/aaronpolhamus/Documents/repos/gensim/gensim/corpora/sharded_corpus.py in <module>()
     32 _default_dtype = float
     33 try:
---> 34     import theano
     35     _default_dtype = theano.config.floatX
     36 except ImportError:

/usr/local/lib/python2.7/site-packages/theano/**init**.pyc in <module>()
     61     object2, utils)
     62 
---> 63 from theano.compile import (
     64     SymbolicInput, In,
     65     SymbolicOutput, Out,

/usr/local/lib/python2.7/site-packages/theano/compile/**init**.py in <module>()
      7         SpecifyShape, specify_shape, register_specify_shape_c_code)
      8 
----> 9 from theano.compile.function_module import *
     10 
     11 from theano.compile.mode import *

/usr/local/lib/python2.7/site-packages/theano/compile/function_module.py in <module>()
     20 from theano.compat import izip
     21 from theano.gof import graph
---> 22 import theano.compile.mode
     23 from theano.compile.io import (
     24     In, SymbolicInput, SymbolicInputKit, SymbolicOutput)

/usr/local/lib/python2.7/site-packages/theano/compile/mode.py in <module>()
     10 import theano
     11 from theano import gof
---> 12 import theano.gof.vm
     13 from theano.configparser import config
     14 from theano.compile.ops import _output_guard

/usr/local/lib/python2.7/site-packages/theano/gof/vm.py in <module>()
    636 
    637 try:
--> 638     from . import lazylinker_c
    639 
    640     class CVM(lazylinker_c.CLazyLinker, VM):

/usr/local/lib/python2.7/site-packages/theano/gof/lazylinker_c.py in <module>()
    124             args = cmodule.GCC_compiler.compile_args()
    125             cmodule.GCC_compiler.compile_str(dirname, code, location=loc,
--> 126                                              preargs=args)
    127             # Save version into the **init**.py file.
    128             init_py = os.path.join(loc, '**init**.py')

/usr/local/lib/python2.7/site-packages/theano/gof/cmodule.pyc in compile_str(module_name, src_code, location, include_dirs, lib_dirs, libs, preargs, py_module, hide_symbols)
   2194             # difficult to read.
   2195             raise Exception('Compilation failed (return status=%s): %s' %
-> 2196                             (status, compile_stderr.replace('\n', '. ')))
   2197         elif config.cmodule.compilation_warning and compile_stderr:
   2198             # Print errors just below the command line.

Didn't anticipate that problems I was having in gensim would ultimately be related to theano, but calling `import theano` reproduced the error. Following the trace from `import gensim` you can see that's where the problem lies. Here's the steps I followed to debug. 

```
sudo pip uninstall theano
sudo pip uninstall gensim 
sudo pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git #..install 'bleeding edge' theano from dev branch. this is the recommended fix for this issue that I've been seeing on the issue boards
sudo pip install --upgrade gensim 
```

Yes, the ShardedCorpus, which is the only place where theano is imported, sometimes gives problems because of theano-related (and gensim-unrelated) issues.

@tmylk when gensim install fails because of theano issues, can we somehow make it clearer that it's due to theano? Point users to check/ reinstall / uninstall theano?

Or how about removing ShardedCorpus from `__init__`, so that it (and therefore theano) are not imported by default by `import gensim`? In other words, users would have to explicitly `from gensim.corpora import ShardedCorpus` to import it.

@tmylk this didn't make it to the release? We really don't want to be importing a beast like theano for no reason.


Resolving patch:
diff --git a/gensim/corpora/__init__.py b/gensim/corpora/__init__.py
--- a/gensim/corpora/__init__.py
+++ b/gensim/corpora/__init__.py
@@ -15,4 +15,3 @@
 from .textcorpus import TextCorpus
 from .ucicorpus import UciCorpus
 from .malletcorpus import MalletCorpus
-from .sharded_corpus import ShardedCorpus


Test patch: