Repository: explosion/spaCy

Issue:
Labels being created multiple times
## How to reproduce the behaviour
Created a custom NER model the cfg file is showing my entities multiple times 

"extra_labels":[
    "PAY",
    "PAY",
    "PAY",
    "PAY",
    "CPTI",
    "CPTI",
    "CPTI",
    "CPTI",
    "CHG_Amts",
    "CHG_Amts",
    "CHG_Amts",
    "CHG_Amts"
  ],

## Your Environment

* Operating System: Windows 7
* Python Version Used:  3.6
* spaCy Version Used: 2.0.18
* Environment Information:



Discussion:
I suspect that This might be due to your data format of list of list as you showed in#3120.

Can you try with list of tuple to see if it helps.

Otherwise share more details on the code you run to get this
My training set is a local json file to read it in I deserialize using the following 

    with open(localFilePath,encoding='utf-8') as f:
         data = json.load(f)

 If I convert to a list of tuples this fails

I re-checked the code  where I add the labels 
 ```
       for _, annotations in trainData:
            for ent in annotations.get('entities'):
                ner.add_label(ent[2])
```

ent[2] = 'PAY' so this seems correct ? 
Have you any examples of a local json file of training data ?

Looks like this is related to #2179. It should be fixed in the latest `spacy-nightly` already – you can install the latest release from pip and rerun your code to check 🙂
I don't know I am thinking that maybe their is something wrong with my representation as the issue in  #3120, seems to suggest their is something going on. 

By any chance have you any examples of a local file of training data being deserialized, it would really help, even better if you have any that are compatible with json.load()


I reworked the format changing the lists to tuples & it made no difference 

Resolving patch:
diff --git a/spacy/errors.py b/spacy/errors.py
--- a/spacy/errors.py
+++ b/spacy/errors.py
@@ -317,6 +317,10 @@ class Errors(object):
     E113 = ("The newly split token can only have one root (head = 0).")
     E114 = ("The newly split token needs to have a root (head = 0)")
     E115 = ("All subtokens must have associated heads")
+    E116 = ("Cannot currently add labels to pre-trained text classifier. Add "
+            "labels before training begins. This functionality was available "
+            "in previous versions, but had significant bugs that led to poor "
+            "performance")
 
 
 @add_codes
diff --git a/spacy/pipeline/entityruler.py b/spacy/pipeline/entityruler.py
--- a/spacy/pipeline/entityruler.py
+++ b/spacy/pipeline/entityruler.py
@@ -86,7 +86,7 @@ def labels(self):
         """
         all_labels = set(self.token_patterns.keys())
         all_labels.update(self.phrase_patterns.keys())
-        return all_labels
+        return tuple(all_labels)
 
     @property
     def patterns(self):


Test patch:
diff --git a/spacy/tests/pipeline/test_pipe_methods.py b/spacy/tests/pipeline/test_pipe_methods.py
--- a/spacy/tests/pipeline/test_pipe_methods.py
+++ b/spacy/tests/pipeline/test_pipe_methods.py
@@ -112,3 +112,15 @@ def test_add_lots_of_pipes(nlp, n_pipes):
 def test_raise_for_invalid_components(nlp, component):
     with pytest.raises(ValueError):
         nlp.add_pipe(component)
+
+
+@pytest.mark.parametrize("component", ["ner", "tagger", "parser", "textcat"])
+def test_pipe_base_class_add_label(nlp, component):
+    label = "TEST"
+    pipe = nlp.create_pipe(component)
+    pipe.add_label(label)
+    if component == "tagger":
+        # Tagger always has the default coarse-grained label scheme
+        assert label in pipe.labels
+    else:
+        assert pipe.labels == (label,)
diff --git a/spacy/tests/regression/test_issue2001-2500.py b/spacy/tests/regression/test_issue2001-2500.py
--- a/spacy/tests/regression/test_issue2001-2500.py
+++ b/spacy/tests/regression/test_issue2001-2500.py
@@ -11,7 +11,6 @@
 from ..util import add_vecs_to_vocab, get_doc
 
 
-@pytest.mark.xfail
 def test_issue2179():
     """Test that spurious 'extra_labels' aren't created when initializing NER."""
     nlp = Italian()
@@ -23,7 +22,7 @@ def test_issue2179():
     nlp2.add_pipe(nlp2.create_pipe("ner"))
     nlp2.from_bytes(nlp.to_bytes())
     assert "extra_labels" not in nlp2.get_pipe("ner").cfg
-    assert nlp2.get_pipe("ner").labels == ["CITIZENSHIP"]
+    assert nlp2.get_pipe("ner").labels == ("CITIZENSHIP",)
 
 
 def test_issue2219(en_vocab):