Repository: explosion/spaCy

Issue:
Split one token into several
## Feature description
We now have a feature to merge tokens into one, implemented in the retokenizer. We're lacking the reverse feature: split one token into several. An API exist in _retokenize, but is not implemented yet.





Discussion:
Here are some implementation notes and questions. 

- The current API doesn't specify any syntactic dependencies for the new tokens. This raises two issues:

1. Where dependencies that pointed to the original token should point to after the split. Giving the root of the new span as a parameter would allow tokens that pointed to the original token to point to the root of the newly created span.
2. There is no way of knowing the dependencies of the newly created tokens (that I can think of). Here too, having a root argument would allow to give that root as a head to all other newly created tokens, and to keep the original dependencies for the root token itself.

- Here is how I would implement the split, given a root argument:
Double doc.c length if necessary (until big enough for all new tokens)
move tokens after the split to create space for the new tokens
Host the tokens in the newly created space
get LexemeC* for all new orths
set token.spacy to False for all non-last splited tokens, and to origToken.spacy for the last token
apply attrs to each subtoken
if origToken.iob == 3/begin, set the first subtoken to 3/begin, and all other subtokens to 1/in
in all other cases subtokens inherit iob from origToken
Adjust all heads with the offset, similar to the merge function
Heads that pointed to the token now point to the root.
All non root subtokens have the root token as head.
Root subtoken inherit dependencies from origToken.
set children from head

Does that sound like it would work? I can implement it, if so.

Merging tokens is simpler than splitting them because we can just iron out the internal structure of the span to be merged, ie. drop internal dependencies and redirect all other incoming and outgoing dependencies to the syntactic head (provided the merged tokens form a proper subtree).
Note that:
1. The dropped internal dependencies can form different topologies, including but not restricted to the one where all merged tokens are direct dependents of a unique head in the span ;
2. Tokens other than the head can also have dependents on their own, which is fine because they become dependents of the merged token.

The reverse operation (split a token) is trickier because we need to create an internal structure between the newly split tokens and stitch them to the surrounding tokens. The implementation you propose effectively imposes a fixed topology, where one of the new tokens is the head of all the other new tokens and receives all in- and out- dependency edges. I assume the (new) internal dependency edges would have a default label.
Ideally, a token splitter should enable the user to explicitly specify the (internal and external) dependency structure they want. The mechanics involved quickly become intricate though, so...
I think the main issue here is the expected use case. Would the user know the complex structure of the splitted tokens dependencies in advance, or would they reparse, or use a simpler stucture most of the time? It could also make sense to not touch the dependencies at all, and to assume the document is either not parsed yet or will get reparsed afterwards. 
@grivaz Sorry for the delay getting to this. Definitely appreciate the help.

I tried to implement token splitting during parsing earlier this year, as I wanted the parser to handle languages like Arabic by jointly predicting the tokenization. I wanted an action which would divide the second token of the buffer. This ended up being a huge mess, so I backed out the changes. Introducing a split method that works prior to parsing is much more feasible.

For the parse tree, I think it would make sense to take array-valued arguments for the heads and dep labels. The `heads` array would specify head offsets for the new tokens, which dictates the parse tree shape.

For instance, `split(doc, 4, 3, heads=[0, -1, -1])` would insert two new tokens at position 4. The head of the new region would be the first token, with the second token attached to it, and the third token attached to the second one.

If we had something like `split(doc, 4, 3, heads=[1, 0, -1]), the head of the new region would be one of the introduced tokens. To make that work, we'll need to repoint the heads that were going into the region, so that they point to this new head.

Otherwise, what you suggest is good. The only thing I'd note is:

> Double doc.c length if necessary (until big enough for all new tokens)

The capacity is in `doc.max_length`, while the current length is `doc.length`. See `doc.push_back` for reference.

Resolving patch:
diff --git a/spacy/errors.py b/spacy/errors.py
--- a/spacy/errors.py
+++ b/spacy/errors.py
@@ -253,6 +253,9 @@ class Errors(object):
     E098 = ("Trying to set conflicting doc.ents: '{span1}' and '{span2}'. A token"
             " can only be part of one entity, so make sure the entities you're "
             "setting don't overlap.")
+    E099 = ("The newly split token can only have one root (head = 0).")
+    E100 = ("The newly split token needs to have a root (head = 0)")
+    E101 = ("All subtokens must have associated heads")
 
 @add_codes
 class TempErrors(object):


Test patch:
diff --git a/spacy/tests/doc/test_doc_spilt.py b/spacy/tests/doc/test_doc_spilt.py
new file mode 100644
--- /dev/null
+++ b/spacy/tests/doc/test_doc_spilt.py
@@ -0,0 +1,114 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from ..util import get_doc
+from ...vocab import Vocab
+from ...tokens import Doc
+from ...tokens import Span
+
+import pytest
+
+
+def test_doc_split(en_tokenizer):
+    text = "LosAngeles start."
+    heads = [1, 1, 0]
+    tokens = en_tokenizer(text)
+    doc = get_doc(tokens.vocab, [t.text for t in tokens], heads=heads)
+
+    assert len(doc) == 3
+    assert len(str(doc)) == 19
+    assert doc[0].head.text == 'start'
+    assert doc[1].head.text == '.'
+
+    with doc.retokenize() as retokenizer:
+        retokenizer.split(doc[0], ["Los", "Angeles"], [1, 0], attrs={'tag':'NNP', 'lemma':'Los Angeles', 'ent_type':'GPE'})
+
+    assert len(doc) == 4
+    assert doc[0].text == 'Los'
+    assert doc[0].head.text == 'Angeles'
+    assert doc[0].idx == 0
+    assert doc[1].idx == 3
+
+    assert doc[1].text == 'Angeles'
+    assert doc[1].head.text == 'start'
+
+    assert doc[2].text == 'start'
+    assert doc[2].head.text == '.'
+
+    assert doc[3].text == '.'
+    assert doc[3].head.text == '.'
+
+    assert len(str(doc)) == 19
+
+def test_split_dependencies(en_tokenizer):
+    text = "LosAngeles start."
+    tokens = en_tokenizer(text)
+    doc = get_doc(tokens.vocab, [t.text for t in tokens])
+    dep1 = doc.vocab.strings.add('amod')
+    dep2 = doc.vocab.strings.add('subject')
+    with doc.retokenize() as retokenizer:
+        retokenizer.split(doc[0], ["Los", "Angeles"], [1, 0], [dep1, dep2])
+
+    assert doc[0].dep == dep1
+    assert doc[1].dep == dep2
+
+
+
+def test_split_heads_error(en_tokenizer):
+    text = "LosAngeles start."
+    tokens = en_tokenizer(text)
+    doc = get_doc(tokens.vocab, [t.text for t in tokens])
+    #Not enough heads
+    with pytest.raises(ValueError):
+        with doc.retokenize() as retokenizer:
+            retokenizer.split(doc[0], ["Los", "Angeles"], [0])
+
+    #Too many heads
+    with pytest.raises(ValueError):
+        with doc.retokenize() as retokenizer:
+            retokenizer.split(doc[0], ["Los", "Angeles"], [1, 1, 0])
+
+    #No token head
+    with pytest.raises(ValueError):
+        with doc.retokenize() as retokenizer:
+            retokenizer.split(doc[0], ["Los", "Angeles"], [1, 1])
+
+    #Several token heads
+    with pytest.raises(ValueError):
+        with doc.retokenize() as retokenizer:
+            retokenizer.split(doc[0], ["Los", "Angeles"], [0, 0])
+
+
+def test_spans_entity_merge_iob():
+    # Test entity IOB stays consistent after merging
+    words = ["abc", "d", "e"]
+    doc = Doc(Vocab(), words=words)
+    doc.ents = [(doc.vocab.strings.add('ent-abcd'), 0, 2)]
+    assert doc[0].ent_iob_ == "B"
+    assert doc[1].ent_iob_ == "I"
+
+    with doc.retokenize() as retokenizer:
+        retokenizer.split(doc[0], ["a", "b", "c"], [1, 1, 0])
+    assert doc[0].ent_iob_ == "B"
+    assert doc[1].ent_iob_ == "I"
+    assert doc[2].ent_iob_ == "I"
+    assert doc[3].ent_iob_ == "I"
+
+def test_spans_sentence_update_after_merge(en_tokenizer):
+    text = "StewartLee is a stand up comedian. He lives in England and loves JoePasquale."
+    heads = [1, 0, 1, 2, -1, -4, -5, 1, 0, -1, -1, -3, -4, 1, -2]
+    deps = ['nsubj', 'ROOT', 'det', 'amod', 'prt', 'attr',
+            'punct', 'nsubj', 'ROOT', 'prep', 'pobj', 'cc', 'conj',
+            'compound', 'punct']
+
+    tokens = en_tokenizer(text)
+    doc = get_doc(tokens.vocab, [t.text for t in tokens], heads=heads, deps=deps)
+    sent1, sent2 = list(doc.sents)
+    init_len = len(sent1)
+    init_len2 = len(sent2)
+    with doc.retokenize() as retokenizer:
+        retokenizer.split(doc[0], ["Stewart", "Lee"], [1, 0])
+        retokenizer.split(doc[14], ["Joe", "Pasquale"], [1, 0])
+    sent1, sent2 = list(doc.sents)
+    assert len(sent1) == init_len + 1
+    assert len(sent2) == init_len2 + 1