Repository: freelawproject/reporters-db (language: python)

Issue:
Add all reporters from Harvard's collection
@harvard-lil put [a big json file together](https://github.com/harvard-lil/reporter-list/blob/master/reporters.json) with all of their reporters. It's pretty great, and contains almost all the information that we have in our reporter database. I did a very rough study of the differences:

It looks like they have about 600 reporters:

```
lil_db = json.load(open('/tmp/reporters.json', 'r'))
len(lil_db)
```

Prints: 

```
600
```

Of those, I wanted to see which we already had, so:

```
from reporter_db import EDITIONS, REPORTERS, VARIATIONS_ONLY
all_keys = EDITIONS.keys() + REPORTERS.keys() + VARIATIONS_ONLY.keys()
found = 0
not_found = 0
for r in lil_db:
    if r['short'] in all_keys:
        found += 1
    else:
        not_found += 1
print "Found in FLP reporter DB: %s\nNot Found in FLP reporter DB: %s" % (found, not_found)
```

Prints: 

```
Found in FLP reporter DB: 315
Not Found in FLP reporter DB: 285
```

Looking into the ones that we're missing, it's a bit of a mish-mash. There are a number of reporters where we lack the abbreviation that LIL is using. This seems to be because one or the other of us isn't using the recommended abbreviation. For these, we'll want to figure out which is correct and use that as the primary, adding the other as a variation.

The other items that are missing seem to be either small or old reporters. Here's a sampling:

```
Rec. T. Warwick (R.I.) Records of the Court of Trials of the Town of Warwick, 1659-1674
Rec. V.A. Ct. (R.I.) Records of the Vice-Admiralty Court of Rhode Island, 1716-1752
R.I. Ct. Rec. Rhode Island Court Records
R.I. Dec. Rhode Island Decisions
Super. Ct. (R.I.) Rhode Island Superior Court Rescripts
Rec. Co. Ch. (S.C.) Records of the court of Chancery of South Carolina, 1671-1779
Strobh. Reports of cases argued and determined in the court of appeals and the court of errors of South Carolina
Dess.Eq. Reports of cases argud and determined in the court of chancery of the state of south carolina
Rich. Eq. Cas. Reports of cases in equity, argued and determined in the court of appeals and court of errors of south carolina
Strobh. Eq. Reports of cases in equity, argued and determined in the court of appeals and court of errors of south carolina
Unrep. Tenn. Cas. Unreported Tennessee Cases
Tenn. Cas. Tennessee Cases with Notes and Annotations
Tenn. Ch. R. Tennessee Chancery Reports
Va. Ch. Dec. Decisions of the Cases in Virgin
```

For these, we'll probably want to simply incorporate them into our reporter database, making sure that we don't already have them under another name or abbreviation.

Doing this in reverse, to see which items we have that LIL does not, I found:

```
Found in LIL DB: 240
Not Found in LIL DB: 201
```

So the numbers are kind of consistently about 50% found/not found. I haven't dug into these because it's harder to go this direction, but I suspect it'll be similar to the above findings.



Discussion / hints:
@jcushman, I just found this early investigation into our two reporters DBs. I'm guessing this is pretty darned done. What do you think? 
Yeah I dunno. You'll see in a PR shortly that I've just been doing a little hand editing to clean up our New York City reporters for example -- there's seven reporters CAP had with made-up names like `Robertsons Super. Ct. Rep.` that should have been `Hall`, `Sandf.`, `Duer`, `Bosw.`, `Rob.`, `Sweeny`, `Jones & S.` as nominatives of `N.Y. Super. Ct.`, which we didn't have as a reporter at all.

So there's probably more stuff like that lingering that you could potentially find through this comparison ... though I don't know if a general issue for the whole haystack is helpful to keep open. 🤷‍♂️

Fix patch:
diff --git a/reporters_db/data/reporters.json b/reporters_db/data/reporters.json
index 34a9b7f..0c82242 100644
--- a/reporters_db/data/reporters.json
+++ b/reporters_db/data/reporters.json
@@ -298,6 +298,24 @@
             "variations": {}
         }
     ],
+    "Abb. Ct. App.": [
+        {
+            "cite_type": "state",
+            "editions": {
+                "Abb. Ct. App.": {
+                    "end": "1869-12-31T00:00:00",
+                    "start": "1807-01-01T00:00:00"
+                }
+            },
+            "mlz_jurisdiction": [
+                "us:ny;court.appeals"
+            ],
+            "name": "Abbott's Court of Appeals Decisions",
+            "variations": {
+                "Abb. Ct. App. Dec.": "Abb. Ct. App."
+            }
+        }
+    ],
     "Abb. N. Cas.": [
         {
             "cite_type": "state",
@@ -359,6 +377,28 @@
             "variations": {}
         }
     ],
+    "Add.": [
+        {
+            "cite_format": "{reporter} {page}",
+            "cite_type": "state",
+            "editions": {
+                "Add.": {
+                    "end": "1799-12-31T00:00:00",
+                    "start": "1791-01-01T00:00:00"
+                }
+            },
+            "examples": [
+                "Graham v. Goudy, Add. 55"
+            ],
+            "mlz_jurisdiction": [
+                "us:pa;supreme.court",
+                "us:pa;superior.court"
+            ],
+            "name": "Reports of cases of the State of Pennsylvania by Alexander Addison",
+            "notes": "Single volume, typically cited without volume number",
+            "variations": {}
+        }
+    ],
     "Adv. S.": [
         {
             "cite_type": "state",
@@ -754,12 +794,16 @@
             "cite_type": "state",
             "editions": {
                 "Am. Samoa": {
-                    "end": null,
+                    "end": "1975-12-31T00:00:00",
                     "start": "1900-01-01T00:00:00"
                 },
                 "Am. Samoa 2d": {
+                    "end": "1999-12-31T00:00:00",
+                    "start": "1978-01-01T00:00:00"
+                },
+                "Am. Samoa 3d": {
                     "end": null,
-                    "start": "1900-01-01T00:00:00"
+                    "start": "1993-01-01T00:00:00"
                 }
             },
             "mlz_jurisdiction": [
@@ -791,6 +835,20 @@
             }
         }
     ],
+    "Am. Tribal Law": [
+        {
+            "cite_type": "state",
+            "editions": {
+                "Am. Tribal Law": {
+                    "end": null,
+                    "start": "1997-01-01T00:00:00"
+                }
+            },
+            "mlz_jurisdiction": [],
+            "name": "West's American Tribal Law Reporter",
+            "variations": {}
+        }
+    ],
     "Amer. L. Rev.": [
         {
             "cite_type": "federal",
@@ -3525,6 +3583,22 @@
             }
         }
     ],
+    "D.C.": [
+        {
+            "cite_type": "state",
+            "editions": {
+                "D.C.": {
+                    "end": "1893-12-31T00:00:00",
+                    "start": "1801-01-01T00:00:00"
+                }
+            },
+            "mlz_jurisdiction": [
+                "us:dc;court.appeals"
+            ],
+            "name": "District of Columbia Reports",
+            "variations": {}
+        }
+    ],
     "Daily Journal DAR": [
         {
             "cite_type": "state",
@@ -8248,6 +8322,22 @@
             "variations": {}
         }
     ],
+    "Mass. L. Rptr.": [
+        {
+            "cite_type": "state",
+            "editions": {
+                "Mass. L. Rptr.": {
+                    "end": null,
+                    "start": "1993-01-01T00:00:00"
+                }
+            },
+            "mlz_jurisdiction": [
+                "us:ma;superior.court"
+            ],
+            "name": "Massachusetts Law Reporter",
+            "variations": {}
+        }
+    ],
     "Mass. Supp.": [
         {
             "cite_type": "state",
@@ -8673,13 +8763,13 @@
             }
         }
     ],
-    "Mills.": [
+    "Mills Surr.": [
         {
             "cite_type": "state",
             "editions": {
-                "Mills.": {
-                    "end": null,
-                    "start": "1750-01-01T00:00:00"
+                "Mills Surr.": {
+                    "end": "1917-12-31T00:00:00",
+                    "start": "1899-01-01T00:00:00"
                 }
             },
             "mlz_jurisdiction": [
@@ -8687,8 +8777,9 @@
             ],
             "name": "Mills' Reports, New York Surrogate Court",
             "variations": {
-                "Mills": "Mills.",
-                "mills": "Mills."
+                "Mills": "Mills Surr.",
+                "Mills.": "Mills Surr.",
+                "mills": "Mills Surr."
             }
         }
     ],
@@ -9537,13 +9628,13 @@
             }
         }
     ],
-    "N.Y. Crim. R.": [
+    "N.Y. Crim.": [
         {
             "cite_type": "state",
             "editions": {
-                "N.Y. Crim. R.": {
-                    "end": null,
-                    "start": "1750-01-01T00:00:00"
+                "N.Y. Crim.": {
+                    "end": "1924-12-31T00:00:00",
+                    "start": "1810-01-01T00:00:00"
                 }
             },
             "mlz_jurisdiction": [
@@ -9551,10 +9642,11 @@
             ],
             "name": "New York Criminal Reports",
             "variations": {
-                "N.Y. Cr. R.": "N.Y. Crim. R.",
-                "N.Y. Cr. Rep.": "N.Y. Crim. R.",
-                "N.Y. Crim. Rep.": "N.Y. Crim. R.",
-                "N.Y.Crim.R.": "N.Y. Crim. R."
+                "N.Y. Cr. R.": "N.Y. Crim.",
+                "N.Y. Cr. Rep.": "N.Y. Crim.",
+                "N.Y. Crim. R.": "N.Y. Crim.",
+                "N.Y. Crim. Rep.": "N.Y. Crim.",
+                "N.Y.Crim.R.": "N.Y. Crim."
             }
         }
     ],
@@ -9588,6 +9680,9 @@
             "mlz_jurisdiction": [],
             "name": "New York State Reporter",
             "variations": {
+                "N.Y. St. R.": "N.Y. St. Rep.",
+                "N.Y. St. Rep'r": "N.Y. St. Rep.",
+                "N.Y. St. Repr.": "N.Y. St. Rep.",
                 "N.Y.St.Rep.": "N.Y. St. Rep.",
                 "St. Rep.": "N.Y. St. Rep.",
                 "St.Rep.": "N.Y. St. Rep."
@@ -10647,7 +10742,13 @@
                 "O.N.P.N.S.": "Ohio N.P. (n.s.)",
                 "Oh.N.P.": "Ohio N.P.",
                 "Oh.N.P.(N.S).": "Ohio N.P. (n.s.)",
-                "Ohio N.P.N.S.": "Ohio N.P. (n.s.)"
+                "Ohio N.P.N.S.": "Ohio N.P. (n.s.)",
+                "Ohio Nisi Prius": "Ohio N.P.",
+                "Ohio Nisi Prius (N. S.)": "Ohio N.P. (n.s.)",
+                "Ohio Nisi Prius (NS)": "Ohio N.P. (n.s.)",
+                "Ohio Nisi Prius Decisions": "Ohio N.P.",
+                "Ohio Nisi Prius Reports (N. S.)": "Ohio N.P. (n.s.)",
+                "Ohio Nisi Prius [N. S.]": "Ohio N.P. (n.s.)"
             }
         }
     ],
@@ -11030,7 +11131,7 @@
                 }
             },
             "mlz_jurisdiction": [
-                "us:c1:pr.d;district.court"
+                "us:pr;supreme.court"
             ],
             "name": "Decisiones de Puerto Rico",
             "variations": {
@@ -11039,6 +11140,24 @@
             }
         }
     ],
+    "P.R. Fed.": [
+        {
+            "cite_type": "federal",
+            "editions": {
+                "P.R. Fed.": {
+                    "end": "1924-12-31T00:00:00",
+                    "start": "1900-01-01T00:00:00"
+                }
+            },
+            "mlz_jurisdiction": [
+                "us:c1:pr.d;district.court"
+            ],
+            "name": "Porto Rico Federal Reports",
+            "variations": {
+                "P.R. Fed. Rep.": "P.R. Fed."
+            }
+        }
+    ],
     "P.R. Offic. Trans.": [
         {
             "cite_type": "state",
@@ -11049,7 +11168,7 @@
                 }
             },
             "mlz_jurisdiction": [
-                "us:c1:pr.d;district.court"
+                "us:pr;supreme.court"
             ],
             "name": "Official Translations of the Opinions of the Supreme Court of Puerto Rico",
             "variations": {}
@@ -11065,7 +11184,7 @@
                 }
             },
             "mlz_jurisdiction": [
-                "us:c1:pr.d;district.court"
+                "us:pr;supreme.court"
             ],
             "name": "Sentencias del Tribunal Supremo de Puerto Rico",
             "variations": {}
@@ -11270,6 +11389,10 @@
                 "Pa. D. & C.4th": {
                     "end": null,
                     "start": "1921-01-01T00:00:00"
+                },
+                "Pa. D. & C.5th": {
+                    "end": null,
+                    "start": "2006-01-01T00:00:00"
                 }
             },
             "mlz_jurisdiction": [
@@ -11277,6 +11400,10 @@
             ],
             "name": "Pennsylvania District and County Reports",
             "variations": {
+                "Pa. D. & C. 2d": "Pa. D. & C.2d",
+                "Pa. D. & C. 3d": "Pa. D. & C.3d",
+                "Pa. D. & C. 4th": "Pa. D. & C.4th",
+                "Pa. D. & C. 5th": "Pa. D. & C.5th",
                 "Pa.Dist.& C.Rep.": "Pa. D. & C.",
                 "Pa.Dist.& Co.": "Pa. D. & C."
             }
@@ -12292,8 +12419,8 @@
             "cite_type": "state",
             "editions": {
                 "S.C. Eq.": {
-                    "end": null,
-                    "start": "1750-01-01T00:00:00"
+                    "end": "1868-12-31T00:00:00",
+                    "start": "1784-01-01T00:00:00"
                 }
             },
             "mlz_jurisdiction": [
@@ -12303,6 +12430,22 @@
             "variations": {}
         }
     ],
+    "S.C.L.": [
+        {
+            "cite_type": "state",
+            "editions": {
+                "S.C.L.": {
+                    "end": "1868-12-31T00:00:00",
+                    "start": "1783-01-01T00:00:00"
+                }
+            },
+            "mlz_jurisdiction": [
+                "us:sc;supreme.court"
+            ],
+            "name": "South Carolina Law Reports",
+            "variations": {}
+        }
+    ],
     "S.D.": [
         {
             "cite_type": "state",
@@ -13184,6 +13327,22 @@
             }
         }
     ],
+    "T.C.A.": [
+        {
+            "cite_type": "state",
+            "editions": {
+                "T.C.A.": {
+                    "end": null,
+                    "start": "1995-01-01T00:00:00"
+                }
+            },
+            "mlz_jurisdiction": [
+                "us:pr;court.appeals"
+            ],
+            "name": "Decisiones del Tribunal de Circuito de Apelaciones de Puerto Rico (Decisions of the Circuit Court of Appeals of Puerto Rico)",
+            "variations": {}
+        }
+    ],
     "T.C.M.": [
         {
             "cite_type": "specialty",
@@ -13658,6 +13817,22 @@
             }
         }
     ],
+    "Thomp. & Cook": [
+        {
+            "cite_type": "state",
+            "editions": {
+                "Thomp. & Cook": {
+                    "end": "1875-12-31T00:00:00",
+                    "start": "1873-01-01T00:00:00"
+                }
+            },
+            "mlz_jurisdiction": [
+                "us:ny;superior.court"
+            ],
+            "name": "Thompson & Cook's Supreme Court Reports",
+            "variations": {}
+        }
+    ],
     "Thompson": [
         {
             "cite_type": "state",


Test patch:
diff --git a/tests.py b/tests.py
index 1c1b735..d4cb9dd 100644
--- a/tests.py
+++ b/tests.py
@@ -210,7 +210,7 @@ def test_fields_tidy(self):
         """
 
         def cleaner(s):
-            return re.sub(r"[^ 0-9a-zA-Z.,\-'&()]", "", s.strip())
+            return re.sub(r"[^ 0-9a-zA-Z.,\-'&()\[\]]", "", s.strip())
 
         msg = "Got bad punctuation in: %s"
         for reporter_abbv, reporter_list in REPORTERS.items():