Repository: internetarchive/openlibrary

Issue:
AssertionError: leader_element.tag == leader_tag on some imports from archive.org
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

AKA OL-WEB-50B from Sentry logging.

> openlibrary/catalog/marc/marc_xml.py in leader at line 76
> AssertionError: leader_element.tag == leader_tag


Triggered by public API imports to /api/import/ia (POST)  by archive.org identifier.

https://github.com/internetarchive/openlibrary/blob/3fcc46d7471a9561a3165e4db6c8de6aa7fc8778/openlibrary/catalog/get_ia.py#L32 is fetching the MARC XML for import.

**TODO:**

- [ ] Correct https://github.com/internetarchive/openlibrary/blob/3fcc46d7471a9561a3165e4db6c8de6aa7fc8778/openlibrary/catalog/marc/marc_xml.py#L71-L72 to pick the `leader` tag rather than the first one (in these error cases it is XML comments ... which is allowed, but indicates there was an issue converting from binary MARC)
- [ ] Switch priority of MARC XML / MARC binary in https://github.com/internetarchive/openlibrary/blob/3fcc46d7471a9561a3165e4db6c8de6aa7fc8778/openlibrary/catalog/get_ia.py#L32  -- try MARC binary first as that is generally the original data source and MARC binary process is the more tested import method in OL.   



### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->

Example archive.org identifiers with MARC XML comments which have triggered this issue (via Sentry):
* exvoto0000didi
* candidahferbolog0000hfer


### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@cclauss 
AssertionError: leader_element.tag == leader_tag on some imports from archive.org
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

AKA OL-WEB-50B from Sentry logging.

> openlibrary/catalog/marc/marc_xml.py in leader at line 76
> AssertionError: leader_element.tag == leader_tag


Triggered by public API imports to /api/import/ia (POST)  by archive.org identifier.

https://github.com/internetarchive/openlibrary/blob/3fcc46d7471a9561a3165e4db6c8de6aa7fc8778/openlibrary/catalog/get_ia.py#L32 is fetching the MARC XML for import.

**TODO:**

- [ ] Correct https://github.com/internetarchive/openlibrary/blob/3fcc46d7471a9561a3165e4db6c8de6aa7fc8778/openlibrary/catalog/marc/marc_xml.py#L71-L72 to pick the `leader` tag rather than the first one (in these error cases it is XML comments ... which is allowed, but indicates there was an issue converting from binary MARC)
- [ ] Switch priority of MARC XML / MARC binary in https://github.com/internetarchive/openlibrary/blob/3fcc46d7471a9561a3165e4db6c8de6aa7fc8778/openlibrary/catalog/get_ia.py#L32  -- try MARC binary first as that is generally the original data source and MARC binary process is the more tested import method in OL.   



### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->

Example archive.org identifiers with MARC XML comments which have triggered this issue (via Sentry):
* exvoto0000didi
* candidahferbolog0000hfer


### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@cclauss 


Discussion:

Resolving patch:
diff --git a/openlibrary/catalog/get_ia.py b/openlibrary/catalog/get_ia.py
--- a/openlibrary/catalog/get_ia.py
+++ b/openlibrary/catalog/get_ia.py
@@ -1,9 +1,7 @@
-import traceback
-import xml.parsers.expat
+import requests
 
 from infogami import config
 from lxml import etree
-import requests
 from time import sleep
 
 from openlibrary.catalog.marc.marc_binary import MarcBinary
@@ -16,7 +14,7 @@
 MAX_MARC_LENGTH = 100000
 
 
-def urlopen_keep_trying(url, headers=None, **kwargs):
+def urlopen_keep_trying(url: str, headers=None, **kwargs):
     """Tries to request the url three times, raises HTTPError if 403, 404, or 416.  Returns a requests.Response"""
     for i in range(3):
         try:
@@ -29,14 +27,11 @@ def urlopen_keep_trying(url, headers=None, **kwargs):
         sleep(2)
 
 
-def get_marc_record_from_ia(identifier):
+def get_marc_record_from_ia(identifier: str) -> MarcBinary | MarcXml | None:
     """
     Takes IA identifiers and returns MARC record instance.
     08/2018: currently called by openlibrary/plugins/importapi/code.py
     when the /api/import/ia endpoint is POSTed to.
-
-    :param str identifier: ocaid
-    :rtype: MarcXML | MarcBinary
     """
     metadata = ia.get_metadata(identifier)
     filenames = metadata['_filenames']
@@ -46,60 +41,17 @@ def get_marc_record_from_ia(identifier):
 
     item_base = f'{IA_DOWNLOAD_URL}{identifier}/'
 
-    # Try marc.xml first
-    if marc_xml_filename in filenames:
-        data = urlopen_keep_trying(item_base + marc_xml_filename).content
-        try:
-            root = etree.fromstring(data)
-            return MarcXml(root)
-        except Exception as e:
-            print("Unable to read MarcXML: %s" % e)
-            traceback.print_exc()
-
-    # If that fails, try marc.bin
+    # Try marc.bin first
     if marc_bin_filename in filenames:
         data = urlopen_keep_trying(item_base + marc_bin_filename).content
         return MarcBinary(data)
 
-
-def files(identifier):
-    url = item_file_url(identifier, 'files.xml')
-    for i in range(5):
-        try:
-            tree = etree.parse(urlopen_keep_trying(url).content)
-            break
-        except xml.parsers.expat.ExpatError:
-            sleep(2)
-    try:
-        tree = etree.parse(urlopen_keep_trying(url).content)
-    except:
-        print("error reading", url)
-        raise
-    assert tree
-    for i in tree.getroot():
-        assert i.tag == 'file'
-        name = i.attrib['name']
-        if name == 'wfm_bk_marc' or name.endswith(
-            ('.dat', '.marc', '.mrc', '.out', '.records.utf8')
-        ):
-            size = i.find('size')
-            if size is not None:
-                yield name, int(size.text)
-            else:
-                yield name, None
-
-
-def get_from_archive(locator):
-    """
-    Gets a single binary MARC record from within an Archive.org
-    bulk MARC item - data only.
-
-    :param str locator: Locator ocaid/filename:offset:length
-    :rtype: str|None
-    :return: Binary MARC data
-    """
-    data, offset, length = get_from_archive_bulk(locator)
-    return data
+    # If that fails, try marc.xml
+    if marc_xml_filename in filenames:
+        data = urlopen_keep_trying(item_base + marc_xml_filename).content
+        root = etree.fromstring(data)
+        return MarcXml(root)
+    return None
 
 
 def get_from_archive_bulk(locator):
@@ -146,42 +98,3 @@ def get_from_archive_bulk(locator):
             else:
                 next_offset = next_length = None
     return data, next_offset, next_length
-
-
-def item_file_url(identifier, ending, host=None, path=None):
-    if host and path:
-        url = f'http://{host}{path}/{identifier}_{ending}'
-    else:
-        url = '{0}{1}/{1}_{2}'.format(IA_DOWNLOAD_URL, identifier, ending)
-    return url
-
-
-def marc_formats(identifier, host=None, path=None):
-    files = {
-        identifier + '_marc.xml': 'xml',
-        identifier + '_meta.mrc': 'bin',
-    }
-    has = {'xml': False, 'bin': False}
-    url = item_file_url(identifier, 'files.xml', host, path)
-    for attempt in range(10):
-        f = urlopen_keep_trying(url)
-        if f is not None:
-            break
-        sleep(10)
-    if f is None:
-        # TODO: log this, if anything uses this code
-        msg = "error reading %s_files.xml" % identifier
-        return has
-    data = f.content
-    try:
-        root = etree.fromstring(data)
-    except:
-        print(('bad:', repr(data)))
-        return has
-    for e in root:
-        name = e.attrib['name']
-        if name in files:
-            has[files[name]] = True
-        if all(has.values()):
-            break
-    return has
diff --git a/openlibrary/catalog/marc/marc_xml.py b/openlibrary/catalog/marc/marc_xml.py
--- a/openlibrary/catalog/marc/marc_xml.py
+++ b/openlibrary/catalog/marc/marc_xml.py
@@ -72,9 +72,10 @@ def leader(self) -> str:
         leader_element = self.record[0]
         if not isinstance(leader_element.tag, str):
             leader_element = self.record[1]
-        assert (
-            leader_element.tag == leader_tag
-        ), f"{leader_element.tag = } != {leader_tag = }"
+        assert leader_element.tag == leader_tag, (
+            'MARC XML is possibly corrupt in conversion. Unexpected non-Leader tag: '
+            f'{leader_element.tag}'
+        )
         return get_text(leader_element)
 
     def read_fields(self, want: list[str]) -> Iterator[tuple[str, str | DataField]]:


Test patch:
diff --git a/openlibrary/tests/catalog/test_get_ia.py b/openlibrary/tests/catalog/test_get_ia.py
--- a/openlibrary/tests/catalog/test_get_ia.py
+++ b/openlibrary/tests/catalog/test_get_ia.py
@@ -83,11 +83,11 @@ class TestGetIA:
         'zweibchersatir01horauoft',
     ]
 
-    @pytest.mark.parametrize('item', xml_items)
+    @pytest.mark.parametrize('item', bin_items)
     def test_get_marc_record_from_ia(self, item, monkeypatch):
         """Tests the method returning MARC records from IA
-        used by the import API. It should return an XML MARC if one exists."""
-        monkeypatch.setattr(get_ia, 'urlopen_keep_trying', return_test_marc_xml)
+        used by the import API. It should return a binary MARC if one exists."""
+        monkeypatch.setattr(get_ia, 'urlopen_keep_trying', return_test_marc_bin)
         monkeypatch.setattr(
             ia,
             'get_metadata',
@@ -97,22 +97,20 @@ def test_get_marc_record_from_ia(self, item, monkeypatch):
         )
         result = get_ia.get_marc_record_from_ia(item)
         assert isinstance(
-            result, MarcXml
-        ), f"{item}: expected instanceof MarcXml, got {type(result)}"
+            result, MarcBinary
+        ), f"{item}: expected instanceof MarcBinary, got {type(result)}"
 
-    @pytest.mark.parametrize('item', bin_items)
+    @pytest.mark.parametrize('item', xml_items)
     def test_no_marc_xml(self, item, monkeypatch):
-        """When no XML MARC is listed in _filenames, the Binary MARC should be fetched."""
-        monkeypatch.setattr(get_ia, 'urlopen_keep_trying', return_test_marc_bin)
+        """When no binary MARC is listed in _filenames, the MARC XML should be fetched."""
+        monkeypatch.setattr(get_ia, 'urlopen_keep_trying', return_test_marc_xml)
         monkeypatch.setattr(
-            ia, 'get_metadata', lambda itemid: {'_filenames': [f'{itemid}_meta.mrc']}
+            ia, 'get_metadata', lambda itemid: {'_filenames': [f'{itemid}_marc.xml']}
         )
         result = get_ia.get_marc_record_from_ia(item)
         assert isinstance(
-            result, MarcBinary
-        ), f"{item}: expected instanceof MarcBinary, got {type(result)}"
-        field_245 = next(result.read_fields(['245']))
-        title = next(field_245[1].get_all_subfields())[1].encode('utf8')
+            result, MarcXml
+        ), f"{item}: expected instanceof MarcXml, got {type(result)}"
 
     @pytest.mark.parametrize('bad_marc', bad_marcs)
     def test_incorrect_length_marcs(self, bad_marc, monkeypatch):