Repository: internetarchive/openlibrary

Issue:
IA Batch Imports missing batch_id
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->
This is related to investigation of why some `ocaids` (i.e. archive.org IDs) seem to be failing to import into Open Library's catalog.

### Evidence / Screenshot (if possible)

Each day, Open Library uses a cron to discover and queue new IA items for import.
It should add to a batch called `new-scans-{yyyymm}`
For the past 3+ months we see these batches getting created in import_batch table, but there are 0 items
However, we see ~880k import_items with `ia` identifiers that belong to a NULL batch.

### Stakeholders
<!-- @ tag stakeholders of this bug -->
cc: @scottbarnes + @cdrini 


Discussion:
On `ol-db1` we expect the Internet Archive monthly batches to be created + populated with items. The batches are being initialized but records are failing to be inserted. Here's a view of the db w/ some queries:

```
ssh -A ol-db1
sudo -u postgres psql
openlibrary=# select * from import_batch order by id desc limit 10;
  id  |             name             | submitter |        submit_time         
------+------------------------------+-----------+----------------------------
 1033 | bwb_daily_pallets_2023-07-10 |           | 2023-07-12 01:00:03.840812
 1032 | bwb_daily_pallets_2023-07-09 |           | 2023-07-11 01:00:03.858262
 1031 | bwb_daily_pallets_2023-07-07 |           | 2023-07-09 01:00:04.019244
 1030 | bwb_daily_pallets_2023-07-06 |           | 2023-07-08 01:00:03.613966
 1029 | bwb_daily_pallets_2023-07-05 |           | 2023-07-07 01:00:03.449763
 1028 | bwb_daily_pallets_2023-07-04 |           | 2023-07-06 01:00:03.99117
 1027 | bwb_daily_pallets_2023-07-03 |           | 2023-07-05 01:00:04.028251
 1026 | bwb_daily_pallets_2023-07-02 |           | 2023-07-04 01:00:03.762564
 1025 | new-scans-202307             |           | 2023-07-02 04:30:21.138042
 1024 | bwb_daily_pallets_2023-06-30 |           | 2023-07-02 01:00:04.480781
(10 rows)
;; we could also use 998 which is new-scans-202306 which also has 0 items
openlibrary=# select count(*) from import_item where batch_id=1025;
 count 
-------
     0
(1 row)
;; Compare this to a batch with items
openlibrary=# select count(*) from import_item where batch_id=1024;
 count 
-------
  2381
(1 row)
```

In terms of code paths, this process is run by our daily cron job on ol-home0 (cron container). We know this is running because the batch is being created. The cron then calls https://github.com/internetarchive/openlibrary/blob/master/scripts/manage-imports.py#L120 (manage-imports). We suspect that either get_candidate_ids or some later step is failing in our process.

The way to test would be (given this is run **run** on ol-home0 via the cron container):

```
docker exec -it -uopenlibrary openlibrary-cron-jobs-1 bash
python
>>> import web
>>> import infogami
>>> from openlibrary.config import load_config
>>> load_config('/olsystem/etc/openlibrary.yml')
>>> infogami._setup()
>>> from infogami import config;
>>> from openlibrary.core.ia import get_candidate_ocaids
>>> date = datetime.date.today() - datetime.timedelta(days=1)
>>> items = get_candidate_ocaids(since_date=date)
>>> from openlibrary.core.imports import Batch, ImportItem
>>> b = Batch.find('new-scans-202307')
>>> b.add_items(items)
2023-07-12 19:24:57 [5378] [openlibrary.imports] [INFO] batch new-scans-202307: adding 8899 items
2023-07-12 19:24:59 [5378] [openlibrary.imports] [INFO] batch new-scans-202307: 8899 items are already present, ignoring...
```
It looks like the items are being added to the import queue but not assigned a `batch_id`:

```
openlibrary=# select * from import_item where batch_id is NULL limit 10;
    id    | batch_id |        import_time         |  status  | error |          ia_id           |       ol_key       | comments |         added_time         | data 
----------+----------+----------------------------+----------+-------+--------------------------+--------------------+----------+----------------------------+------
 28637414 |          | 2023-07-12 04:43:21.047336 | modified |       | proceedingsof2000000nati | /books/OL18209851M |          | 2023-07-12 04:30:21.199662 | 
 28637489 |          | 2023-07-12 04:43:20.414387 | modified |       | museumoriginsrea0000unse | /books/OL16522706M |          | 2023-07-12 04:30:21.199662 | 
 28637413 |          | 2023-07-12 04:43:18.045499 | modified |       | mexicohistorical0000unse | /books/OL12368267M |          | 2023-07-12 04:30:21.199662 | 
 28637488 |          | 2023-07-12 04:43:17.584793 | modified |       | isbn_9781597180603       | /books/OL24081314M |          | 2023-07-12 04:30:21.199662 | 
 28637539 |          | 2023-07-12 04:43:17.077424 | modified |       | morehousingmoref0000unse | /books/OL1558187M  |          | 2023-07-12 04:30:21.199662 | 
 28637412 |          | 2023-07-12 04:43:16.43972  | modified |       | isbn_9780472068227       | /books/OL3554935M  |          | 2023-07-12 04:30:21.199662 | 
 28637538 |          | 2023-07-12 04:43:15.695007 | modified |       | engineeringtherm0000pete | /books/OL24013582M |          | 2023-07-12 04:30:21.199662 | 
 28637411 |          | 2023-07-12 04:43:14.581213 | modified |       | isbn_9784431703037       | /books/OL9701227M  |          | 2023-07-12 04:30:21.199662 | 
 28637514 |          | 2023-07-12 04:43:14.562136 | modified |       | translatingstati0000coxv | /books/OL27236337M |          | 2023-07-12 04:30:21.199662 | 
 28637559 |          | 2023-07-12 04:43:14.556606 | modified |       | hamradiohorizons0000odel | /books/OL1751075M  |          | 2023-07-12 04:30:21.199662 | 
(10 rows)

openlibrary=# select count(*) from import_item where batch_id is NULL;
  count  
---------
 1189207
(1 row)
```

We probably want to use the `import_time` to retroactively fix the missing batch_ids and figure out what this is happening moving forward.
Is the problem near here? Ran out of time, off to you @scottbarnes!

https://github.com/internetarchive/openlibrary/blob/master/scripts/manage-imports.py#L139C1-L146C66

Resolving patch:
diff --git a/openlibrary/core/imports.py b/openlibrary/core/imports.py
--- a/openlibrary/core/imports.py
+++ b/openlibrary/core/imports.py
@@ -58,7 +58,7 @@ def dedupe_items(self, items):
 
     def normalize_items(self, items):
         return [
-            {'ia_id': item}
+            {'batch_id': self.id, 'ia_id': item}
             if type(item) is str
             else {
                 'batch_id': self.id,


Test patch:
diff --git a/openlibrary/tests/core/test_imports.py b/openlibrary/tests/core/test_imports.py
--- a/openlibrary/tests/core/test_imports.py
+++ b/openlibrary/tests/core/test_imports.py
@@ -1,71 +1,102 @@
+import pytest
+from typing import Final
 import web
 
 from openlibrary.core.db import get_db
-from openlibrary.core.imports import ImportItem
+from openlibrary.core.imports import Batch, ImportItem
+
+
+IMPORT_ITEM_DDL: Final = """
+CREATE TABLE import_item (
+    id serial primary key,
+    batch_id integer,
+    status text default 'pending',
+    error text,
+    ia_id text,
+    data text,
+    ol_key text,
+    comments text,
+    UNIQUE (batch_id, ia_id)
+);
+"""
+
+IMPORT_BATCH_DDL: Final = """
+CREATE TABLE import_batch (
+    id integer primary key,
+    name text,
+    submitter text,
+    submit_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+"""
+
+IMPORT_ITEM_DATA = [
+    {
+        'id': 1,
+        'batch_id': 1,
+        'ia_id': 'unique_id_1',
+    },
+    {
+        'id': 2,
+        'batch_id': 1,
+        'ia_id': 'unique_id_2',
+    },
+    {
+        'id': 3,
+        'batch_id': 2,
+        'ia_id': 'unique_id_1',
+    },
+]
+
+
+@pytest.fixture(scope="module")
+def setup_item_db():
+    web.config.db_parameters = {'dbn': 'sqlite', 'db': ':memory:'}
+    db = get_db()
+    db.query(IMPORT_ITEM_DDL)
+    yield db
+    db.query('delete from import_item;')
+
+
+@pytest.fixture()
+def import_item_db(setup_item_db):
+    setup_item_db.multiple_insert('import_item', IMPORT_ITEM_DATA)
+    yield setup_item_db
+    setup_item_db.query('delete from import_item;')
 
 
 class TestImportItem:
-    IMPORT_ITEM_DDL = """
-    CREATE TABLE import_item (
-        id serial primary key,
-        batch_id integer,
-        status text default 'pending',
-        error text,
-        ia_id text,
-        data text,
-        ol_key text,
-        comments text,
-        UNIQUE (batch_id, ia_id)
-    );
-    """
-
-    IMPORT_ITEM_DATA = [
-        {
-            'id': 1,
-            'batch_id': 1,
-            'ia_id': 'unique_id_1',
-        },
-        {
-            'id': 2,
-            'batch_id': 1,
-            'ia_id': 'unique_id_2',
-        },
-        {
-            'id': 3,
-            'batch_id': 2,
-            'ia_id': 'unique_id_1',
-        },
-    ]
-
-    @classmethod
-    def setup_class(cls):
-        web.config.db_parameters = {'dbn': 'sqlite', 'db': ':memory:'}
-        db = get_db()
-        db.query(cls.IMPORT_ITEM_DDL)
-
-    @classmethod
-    def teardown_class(cls):
-        db = get_db()
-        db.query('delete from import_item;')
-
-    def setup_method(self):
-        self.db = get_db()
-        self.db.multiple_insert('import_item', self.IMPORT_ITEM_DATA)
-
-    def teardown_method(self):
-        self.db.query('delete from import_item;')
-
-    def test_delete(self):
-        assert len(list(self.db.select('import_item'))) == 3
+    def test_delete(self, import_item_db):
+        assert len(list(import_item_db.select('import_item'))) == 3
 
         ImportItem.delete_items(['unique_id_1'])
-        assert len(list(self.db.select('import_item'))) == 1
+        assert len(list(import_item_db.select('import_item'))) == 1
 
-    def test_delete_with_batch_id(self):
-        assert len(list(self.db.select('import_item'))) == 3
+    def test_delete_with_batch_id(self, import_item_db):
+        assert len(list(import_item_db.select('import_item'))) == 3
 
         ImportItem.delete_items(['unique_id_1'], batch_id=1)
-        assert len(list(self.db.select('import_item'))) == 2
+        assert len(list(import_item_db.select('import_item'))) == 2
 
         ImportItem.delete_items(['unique_id_1'], batch_id=2)
-        assert len(list(self.db.select('import_item'))) == 1
+        assert len(list(import_item_db.select('import_item'))) == 1
+
+
+@pytest.fixture(scope="module")
+def setup_batch_db():
+    web.config.db_parameters = {'dbn': 'sqlite', 'db': ':memory:'}
+    db = get_db()
+    db.query(IMPORT_BATCH_DDL)
+    yield db
+    db.query('delete from import_batch;')
+
+
+class TestBatchItem:
+    def test_add_items_legacy(self, setup_batch_db):
+        """This tests the legacy format of list[str] for items."""
+        legacy_items = ["ocaid_1", "ocaid_2"]
+        batch = Batch.new("test-legacy-batch")
+        result = batch.normalize_items(legacy_items)
+        assert result == [
+            {'batch_id': 1, 'ia_id': 'ocaid_1'},
+            {'batch_id': 1, 'ia_id': 'ocaid_2'},
+        ]