Repository: freelawproject/eyecite

Issue:
Parsing error for citations with defendant 'Thompson'
In issue [#3924](https://github.com/freelawproject/courtlistener/issues/3924), we identified a bug in Eyecite's parsing method when the defendant's last name is 'Thompson'.

For example, for the citation `'Shapiro v. Thompson, 394 U. S. 618'`:

- Expected output: `volume: 394, reporter: 'U.S.', page: '618'`
- Actual output: `volume: None, reporter: 'Thompson', page: '394'`

Other examples of inputs that are incorrectly parsed are: `Adams v. Thompson, 560 F. Supp. 894` and `Mozena v. Thompson, 44 A.2d 276`.


I've been using the first example to debug this issue, and noticed that Eyecite identifies two tokens within the input string: ["Thompson's Unreported Cases (TN)"](https://github.com/freelawproject/reporters-db/blob/d3c57f01e97f5e46d6a6c6cd4ebab46aeb5e7b65/reporters_db/data/reporters.json#L25080-L25106) and ["United States Supreme Court Reports."](https://github.com/freelawproject/reporters-db/blob/d3c57f01e97f5e46d6a6c6cd4ebab46aeb5e7b65/reporters_db/data/reporters.json#L25405-L25455). The problem arises because these tokens overlap (both include "394") and Eyecite's [tokenize](https://github.com/freelawproject/eyecite/blob/3e7836a62e8bd1c6e60eb95a5dc47fded74a9283/eyecite/tokenizers.py#L293-L329) method prioritizes the rightmost token when encountering overlaps, leading to this results.

Discussion:
@mlissner: Any idea how easy this is to solve so that it identifies each? @mlissner: Per discussion today, seems to be happening when citations appear to overlap. The simple solution here is to find both citations that overlap and then filter out the one that's incomplete. @quevon24: The problem here is that we have Thompson as a nominative reporter (optional volume), that's why it is detected as a citation.

We could detect the overlap and see which one is incomplete but according to the reporters-db Thompson regex, `Thompson , 394` is a valid citation, we get two valid citations: `Thompson , 394` and `394 U. S. 618`

This can be replicated with other nominative reporters like:

- Foo v. Cooke (Tennessee Reports, Cooke)
- Foo v. Holmes (Holmes Circuit Court Reports (US))
- Foo v. McCahon (Kansas Reports, McCahon)
- Foo v. Olcott (Olcott's Admiralty Reports)
- Foo v. Taney (Taney's United States (US) Circuit Court Reports)

@mlissner: Seems like some post processing could detect and remedy this. The Thompson one may be technically complete, but it's the same data. So maybe: If there's overlap and the data overlaps too, then....@flooie: @mlissner I like your solution - 

if there is an overlapping span - we can check if one is a complete citation and if one is nominative and choose the complete citation.  

Solution patch:
diff --git a/CHANGES.md b/CHANGES.md
index fe69f23..a03c6f0 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -11,7 +11,8 @@ Changes:
 - None
 
 Fixes:
-- None
+- Prefer the other full citation on overlap with nominative reporter 
+  citations #237
 
 
 ## Current
diff --git a/eyecite/tokenizers.py b/eyecite/tokenizers.py
index ba9ea7d..4b29430 100644
--- a/eyecite/tokenizers.py
+++ b/eyecite/tokenizers.py
@@ -55,6 +55,46 @@ from eyecite.regexes import (
 EXTRACTORS = []
 EDITIONS_LOOKUP = defaultdict(list)
 
+NOMINATIVE_REPORTER_NAMES = {
+    "Thompson",
+    "Cooke",
+    "Holmes",
+    "Olcott",
+    "Chase",
+    "Gilmer",
+    "Bee",
+    "Deady",
+    "Taney",
+}
+
+
+def token_is_from_nominative_reporter(token: Token) -> bool:
+    """Returns true if the token is a citation from a nominative reporter
+
+    Cleaner way to do this would be via an attribute or named group from
+    reporters-db. However; this tagging is currently not complete, so we can
+    use a list of the most problematic names `NOMINATIVE_REPORTER_NAMES`
+
+    ```
+    volume_nominative = token.groups.get("volume_nominative", False)
+    reporter_nominative = token.groups.get("reporter_nominative", False)
+    token.exact_editions[0].reporter.name
+    return volume_nominative is None or volume_nominative
+        or reporter_nominative is None or reporter_nominative
+    ```
+
+    :param token: the token
+    :return: True if the token has a `volume_nominative` group, even if there
+        was no match; False if it didn't have the group
+    """
+    if not isinstance(token, CitationToken):
+        return False
+    if token.exact_editions:
+        name = token.exact_editions[0].reporter.short_name
+    else:
+        name = token.variation_editions[0].reporter.short_name
+    return name in NOMINATIVE_REPORTER_NAMES
+
 
 def _populate_reporter_extractors():
     """Populate EXTRACTORS and EDITIONS_LOOKUP."""
@@ -313,8 +353,19 @@ class Tokenizer:
                 if merged:
                     continue
             if offset > token.start:
-                # skip overlaps
-                continue
+                if (
+                    last_token
+                    and isinstance(token, CitationToken)
+                    and token_is_from_nominative_reporter(last_token)
+                ):
+                    # if a token has overlapping matches between a nominative
+                    # reporter and another type of case citation, prefer the
+                    # other case citation. See #221 and #174
+                    citation_tokens.pop(-1)
+                    all_tokens.pop(-1)
+                else:
+                    # skip overlaps
+                    continue
             if offset < token.start:
                 # capture plain text before each match
                 self.append_text(all_tokens, text[offset : token.start])
@@ -326,6 +377,7 @@ class Tokenizer:
         # capture plain text after final match
         if offset < len(text):
             self.append_text(all_tokens, text[offset:])
+
         return all_tokens, citation_tokens
 
     def get_extractors(self, text: str):


Test patch:
diff --git a/tests/test_FindTest.py b/tests/test_FindTest.py
index 17e28af..51ba4ef 100644
--- a/tests/test_FindTest.py
+++ b/tests/test_FindTest.py
@@ -878,6 +878,55 @@ class FindTest(TestCase):
         ]
         self.run_test_pairs(test_pairs, "Disambiguation")
 
+    def test_nominative_reporter_overlaps(self):
+        """Can we parse a full citation where a name looks like a nominative
+        reporter?"""
+        pairs = [
+            (
+                "In re Cooke, 93 Wn. App. 526, 529",
+                case_citation(volume="93", reporter="Wn. App.", page="526"),
+            ),
+            (
+                "Shapiro v. Thompson, 394 U. S. 618",
+                case_citation(volume="394", reporter="U. S.", page="618"),
+            ),
+            (
+                "MacArdell v. Olcott, 82 N.E. 161",
+                case_citation(volume="82", reporter="N.E.", page="161"),
+            ),
+            (
+                "Connecticut v. Holmes, 221 A.3d 407",
+                case_citation(volume="221", reporter="A.3d", page="407"),
+            ),
+            (
+                "Kern v Taney, 11 Pa. D. & C.5th 558 [2010])",
+                case_citation(
+                    volume="11", reporter="Pa. D. & C.5th", page="558"
+                ),
+            ),
+            (
+                "Ellenburg v. Chase, 2004 MT 66",
+                case_citation(volume="2004", reporter="MT", page="66"),
+            ),
+            (
+                "Gilmer, 500 U.S. at 25;",
+                case_citation(
+                    volume="500", reporter="U. S.", page="25", short=True
+                ),
+            ),
+            (
+                "Bison Bee, 778 F. 13 App’x at 73.",
+                case_citation(volume="778", reporter="F.", page="13"),
+            ),
+        ]
+        for cite_string, cite_object in pairs:
+            parsed_cite = get_citations(cite_string)[0]
+            self.assertEqual(
+                parsed_cite,
+                cite_object,
+                f"Nominative reporters getting in the way of parsing: {parsed_cite}",
+            )
+
     def test_custom_tokenizer(self):
         extractors = []
         for e in EXTRACTORS: