Repository: internetarchive/openlibrary

Issue:
Sanity check publication date in MARC 008 field
This IA volume: https://archive.org/details/aroundworldine2010vern
got imported here: This IA volume: https://archive.org/details/aroundworldine2010vern
with 1873 as the publication date instead of the correct 2010.

The MARC documentation is not the best here, so I can't tell if this is a bad record or we're handling it incorrectly, but I think we can improve the handling in either case.

MARC record: https://openlibrary.org/show-records/ia:aroundworldine2010vern
008 field `008 100816r18732010tnu           000 1 eng d`
260 field `260    $aFranklin, Tenn. :$bDalmatian Press,$c2010.`
MARC documentation for 008: https://www.loc.gov/marc/bibliographic/bd008a.html
position 06 == 'r' means interpreted Date 1 / Date2 (ie "18732010") as "Reprint/reissue date and original date" (presumably in that order?)

I'm guessing that this MARC record has the dates in the 008 backwards (ie value in Date 2 should be in Date 1), but even if so, we can do some sanity checks here:

- Make sure that the republication date comes after the original publication date
- Make sure that the publication date in 260$c, if it exists, matches the publication date from 008
Sanity check publication date in MARC 008 field
This IA volume: https://archive.org/details/aroundworldine2010vern
got imported here: This IA volume: https://archive.org/details/aroundworldine2010vern
with 1873 as the publication date instead of the correct 2010.

The MARC documentation is not the best here, so I can't tell if this is a bad record or we're handling it incorrectly, but I think we can improve the handling in either case.

MARC record: https://openlibrary.org/show-records/ia:aroundworldine2010vern
008 field `008 100816r18732010tnu           000 1 eng d`
260 field `260    $aFranklin, Tenn. :$bDalmatian Press,$c2010.`
MARC documentation for 008: https://www.loc.gov/marc/bibliographic/bd008a.html
position 06 == 'r' means interpreted Date 1 / Date2 (ie "18732010") as "Reprint/reissue date and original date" (presumably in that order?)

I'm guessing that this MARC record has the dates in the 008 backwards (ie value in Date 2 should be in Date 1), but even if so, we can do some sanity checks here:

- Make sure that the republication date comes after the original publication date
- Make sure that the publication date in 260$c, if it exists, matches the publication date from 008


Discussion:
@hornc I added your personal label as I think it's relevant. Feel free to remove it.
https://github.com/internetarchive/openlibrary-librarians/issues/1 contains some other examples of bad `008` dates where the given date is `9999` and the `260$c` field contains the correct date. The value `9999` appears to be reserved for use with serials, so these MARC records are incorrectly coded, but we should still handle them better.
@hornc I added your personal label as I think it's relevant. Feel free to remove it.
https://github.com/internetarchive/openlibrary-librarians/issues/1 contains some other examples of bad `008` dates where the given date is `9999` and the `260$c` field contains the correct date. The value `9999` appears to be reserved for use with serials, so these MARC records are incorrectly coded, but we should still handle them better.

Resolving patch:
diff --git a/openlibrary/catalog/marc/html.py b/openlibrary/catalog/marc/html.py
--- a/openlibrary/catalog/marc/html.py
+++ b/openlibrary/catalog/marc/html.py
@@ -21,6 +21,7 @@ def subfields(line):
 class html_record:
     def __init__(self, data):
         assert len(data) == int(data[:5])
+        self.data = data
         self.record = Record(data)
         self.leader = self.record.leader
 
diff --git a/openlibrary/catalog/marc/parse.py b/openlibrary/catalog/marc/parse.py
--- a/openlibrary/catalog/marc/parse.py
+++ b/openlibrary/catalog/marc/parse.py
@@ -327,9 +327,13 @@ def read_languages(rec: MarcBase, lang_008: str | None = None) -> list[str]:
 
 
 def read_pub_date(rec: MarcBase) -> str | None:
+    """
+    Read publish date from 260$c.
+    """
+
     def publish_date(s: str) -> str:
         date = s.strip('[]')
-        if date.lower() == 'n.d.':  # No date
+        if date.lower() in ('n.d.', 's.d.'):  # No date
             date = '[n.d.]'
         return remove_trailing_number_dot(date)
 
@@ -667,15 +671,19 @@ def read_edition(rec: MarcBase) -> dict[str, Any]:
             raise BadMARC("'008' field must not be blank")
         publish_date = f[7:11]
 
-        if re_date.match(publish_date) and publish_date != '0000':
-            edition["publish_date"] = publish_date
-        if f[6] == 't':
-            edition["copyright_date"] = f[11:15]
+        if re_date.match(publish_date) and publish_date not in ('0000', '9999'):
+            edition['publish_date'] = publish_date
+        if f[6] == 'r' and f[11:15] > publish_date:
+            # Incorrect reprint date order
+            update_edition(rec, edition, read_pub_date, 'publish_date')
+        elif f[6] == 't':  # Copyright date
+            edition['copyright_date'] = f[11:15]
+        if 'publish_date' not in edition:  # Publication date fallback to 260$c
+            update_edition(rec, edition, read_pub_date, 'publish_date')
         publish_country = f[15:18]
         if publish_country not in ('|||', '   ', '\x01\x01\x01', '???'):
-            edition["publish_country"] = publish_country.strip()
-        languages = read_languages(rec, lang_008=f[35:38].lower())
-        if languages:
+            edition['publish_country'] = publish_country.strip()
+        if languages := read_languages(rec, lang_008=f[35:38].lower()):
             edition['languages'] = languages
     elif handle_missing_008:
         update_edition(rec, edition, read_languages, 'languages')


Test patch:
diff --git a/openlibrary/catalog/marc/tests/test_data/bin_expect/test-publish-sn-sl-nd.json b/openlibrary/catalog/marc/tests/test_data/bin_expect/test-publish-sn-sl-nd.json
--- a/openlibrary/catalog/marc/tests/test_data/bin_expect/test-publish-sn-sl-nd.json
+++ b/openlibrary/catalog/marc/tests/test_data/bin_expect/test-publish-sn-sl-nd.json
@@ -15,6 +15,7 @@
   "publishers": [
     "[s.n.]"
   ],
+  "publish_date": "[n.d.]",
   "publish_places": [
     "[s.l.]"
   ]
diff --git a/openlibrary/catalog/marc/tests/test_data/bin_input/9999_sd_dates.mrc b/openlibrary/catalog/marc/tests/test_data/bin_input/9999_sd_dates.mrc
new file mode 100644
--- /dev/null
+++ b/openlibrary/catalog/marc/tests/test_data/bin_input/9999_sd_dates.mrc
@@ -0,0 +1 @@
+00419nam a22001695a 4500001001200000005001700012008004100029035001400070100003800084245002600122260004200148300001100190546001300201899001500214988001300229906000700242012717654-320110315150330.0110301s9999    pk            000 0 urd d  a(PkLaDAR)0 aʻAbd-ul-Qayyum Tahir Malihabadi.10aIfadat-e homiyopethi.  aLahore :bMaktabah-e Daniyal,c[s.d.]  a150 p.  aIn Urdu.  a415_565960  a20110315  0MH
\ No newline at end of file
diff --git a/openlibrary/catalog/marc/tests/test_data/bin_input/9999_with_correct_date_in_260.mrc b/openlibrary/catalog/marc/tests/test_data/bin_input/9999_with_correct_date_in_260.mrc
new file mode 100644
--- /dev/null
+++ b/openlibrary/catalog/marc/tests/test_data/bin_input/9999_with_correct_date_in_260.mrc
@@ -0,0 +1 @@
+00436nam a22001695a 4500001001200000005001700012008004100029035001400070100003700084245003600121260004800157300001100205546001500216899001500231988001300246906000700259012716825-720110315150330.0110301s9999    pk            000 0 urd d  a(PkLaDAR)0 aShah Moʻin-al-Din Ahmad Nadavi.10aʻAlimi aman jo Islami manshur.  aHyderabad :bSindh National Academy,c2003.  a236 p.  aIn Sindhi.  a415_565960  a20110315  0MH
\ No newline at end of file
diff --git a/openlibrary/catalog/marc/tests/test_data/bin_input/reprint_date_wrong_order.mrc b/openlibrary/catalog/marc/tests/test_data/bin_input/reprint_date_wrong_order.mrc
new file mode 100644
--- /dev/null
+++ b/openlibrary/catalog/marc/tests/test_data/bin_input/reprint_date_wrong_order.mrc
@@ -0,0 +1 @@
+01299cam  2200289Ka 4500001001300000003000600013005001700019008004100036040001800077020002500095020002200120035002100142100003000163245005200193246003200245250001600277260004700293300002500340440003000365500023100395520023800626650003900864655003100903700002100934994001200955948004200967ocn656308391OCoLC20100924175141.0100816r18732010tnu           000 1 eng d  aLODcLODdLOD  a9781403793966 (pbk.)  a1403793964 (pbk.)  a(OCoLC)6563083911 aVerne, Jules,d1828-1905.10aAround the world in eighty days /cJules Verne.3 aAround the world in 80 days  aUnabridged.  aFranklin, Tenn. :bDalmatian Press,c2010.  axi, 210 p. ;c21 cm. 0aDalmatian Press Classics.  a"Dalmatian Press Classics offer enjoyable editions of greats of literature. The unabridged texts are enhanced with helpful footnotes. Punctuation and spelling have been updated to conform to modern American usage."-t.p. verso.  a"Phileas Fogg's bet that he can span the world in eighty days takes him by train, ship, sled, and elephant across ninettenth-century India, China, and America in one of the wittiest, most enjoyable classics of all time."--Back cover. 0aVoyages around the worldvFiction. 0aAdventure stories, French.1 aKnight, Kathryn.  aZ0bPMR  hNO HOLDINGS IN PMR - 1 OTHER HOLDINGS
\ No newline at end of file
diff --git a/openlibrary/catalog/marc/tests/test_data/xml_expect/soilsurveyrepor00statgoog.json b/openlibrary/catalog/marc/tests/test_data/xml_expect/soilsurveyrepor00statgoog.json
--- a/openlibrary/catalog/marc/tests/test_data/xml_expect/soilsurveyrepor00statgoog.json
+++ b/openlibrary/catalog/marc/tests/test_data/xml_expect/soilsurveyrepor00statgoog.json
@@ -21,6 +21,7 @@
       "entity_type": "org"
     }
   ],
+  "publish_date": "[n.d.]",
   "publish_places": [
     "Ames"
   ],
diff --git a/openlibrary/catalog/marc/tests/test_parse.py b/openlibrary/catalog/marc/tests/test_parse.py
--- a/openlibrary/catalog/marc/tests/test_parse.py
+++ b/openlibrary/catalog/marc/tests/test_parse.py
@@ -80,6 +80,12 @@
     'test-publish-sn-sl-nd.mrc',
 ]
 
+date_tests = [  # MARC, expected publish_date
+    ('9999_sd_dates.mrc', '[n.d.]'),
+    ('reprint_date_wrong_order.mrc', '2010'),
+    ('9999_with_correct_date_in_260.mrc', '2003'),
+]
+
 TEST_DATA = Path(__file__).with_name('test_data')
 
 
@@ -151,6 +157,13 @@ def test_raises_no_title(self):
         with pytest.raises(NoTitle):
             read_edition(rec)
 
+    @pytest.mark.parametrize('marcfile,expect', date_tests)
+    def test_dates(self, marcfile, expect):
+        filepath = TEST_DATA / 'bin_input' / marcfile
+        rec = MarcBinary(filepath.read_bytes())
+        edition = read_edition(rec)
+        assert edition['publish_date'] == expect
+
 
 class TestParse:
     def test_read_author_person(self):