Repository: explosion/spaCy

Issue:
Adding new Language class without modifying the core library code
Will it be better if spaCy will provide a more extensive support for Language classes defined outside the library source code?

According to the [documentation](http://spacy.io/usage/adding-languages) in order to add a new Language class properly you will need to **modify the library's code**. I think this reduces spaCy's flexibility/extensibility and adds some inconvenience in mantaining user defined Language classes.

Currently if a Language class is defined outside spaCy it will work in some cases but in some cases including some CLI commands and getting noun_chunks it will fail probably because of relying on `util.get_lang_class()` function which only checks for language data within the spaCy library.

In some cases like CLI commands it is not very difficult to create separate custom commands as a work-around but for core functionalities like noun_chunks generation ([_get_chunker](https://github.com/explosion/spaCy/blob/f29c3925eeba4b3720412f490bdd2b2991303b3c/spacy/tokens/doc.pyx#L71)) it seems there is no clear alternative.
Are there other alternative to allow getting noun_chunks when using Language class defined outside spacy directory?

Thank you very much for creating, sharing and mantaining spaCy.

## Info about spaCy

* **spaCy version:** 2.0.5
* **Platform:** Linux-4.4.0-93-generic-x86_64-with-Ubuntu-14.04-trusty
* **Python version:** 3.6.3



Discussion:
~This is now addressed by #2348, which adds entry-point hooks.~

Ah, actually we still need entry points for language classes. We only have this for factories currently.

Resolving patch:
diff --git a/spacy/language.py b/spacy/language.py
--- a/spacy/language.py
+++ b/spacy/language.py
@@ -28,7 +28,7 @@
 from .lang.tokenizer_exceptions import TOKEN_MATCH
 from .lang.tag_map import TAG_MAP
 from .lang.lex_attrs import LEX_ATTRS, is_stop
-from .errors import Errors, Warnings, user_warning
+from .errors import Errors
 from . import util
 from . import about
 
@@ -146,9 +146,6 @@ def __init__(
         RETURNS (Language): The newly constructed object.
         """
         user_factories = util.get_entry_points("spacy_factories")
-        for factory in user_factories.keys():
-            if factory in self.factories:
-                user_warning(Warnings.W009.format(name=factory))
         self.factories.update(user_factories)
         self._meta = dict(meta)
         self._path = None
diff --git a/spacy/util.py b/spacy/util.py
--- a/spacy/util.py
+++ b/spacy/util.py
@@ -43,6 +43,11 @@ def get_lang_class(lang):
     RETURNS (Language): Language class.
     """
     global LANGUAGES
+    # Check if an entry point is exposed for the language code
+    entry_point = get_entry_point("spacy_languages", lang)
+    if entry_point is not None:
+        LANGUAGES[lang] = entry_point
+        return entry_point
     if lang not in LANGUAGES:
         try:
             module = importlib.import_module(".lang.%s" % lang, "spacy")
@@ -230,6 +235,19 @@ def get_entry_points(key):
     return result
 
 
+def get_entry_point(key, value):
+    """Check if registered entry point is available for a given name and
+    load it. Otherwise, return None.
+
+    key (unicode): Entry point name.
+    value (unicode): Name of entry point to load.
+    RETURNS: The loaded entry point or None.
+    """
+    for entry_point in pkg_resources.iter_entry_points(key):
+        if entry_point.name == value:
+            return entry_point.load()
+
+
 def is_in_jupyter():
     """Check if user is running spaCy from a Jupyter notebook by detecting the
     IPython kernel. Mainly used for the displaCy visualizer.


Test patch: