Repository: WordPress/openverse-api

Issue:
Index update fails on missing deleted columns
## Description
<!-- Concisely describe the bug. -->
When attempting to run `UPDATE_INDEX`, the task fails because the server is expecting a `deleted` column to exist within the results. This column is added during the `INGEST_UPSTREAM` reindexing, but not during the `UPDATE_INDEX` step.

https://github.com/WordPress/openverse-api/blob/main/ingestion_server/ingestion_server/indexer.py#L397-L420

## Reproduction
<!-- Provide detailed steps to reproduce the bug. -->
1. Attempt to run the `UPDATE_INDEX` task

## Exception
<!-- Add screenshots to show the problem; or delete the section entirely. -->
```
ingestion_server_1  | 2021-11-18 20:02:31,890 INFO indexer.py:271 - PSQL indexer down: batch_size=200000, downloaded_per_second=32246.160138554646
ingestion_server_1  | Process Task-2:
ingestion_server_1  | Traceback (most recent call last):
ingestion_server_1  |   File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
ingestion_server_1  |     self.run()
ingestion_server_1  |   File "/ingestion_server/ingestion_server/tasks.py", line 108, in run
ingestion_server_1  |     indexer.update(self.model, self.since_date)
ingestion_server_1  |   File "/ingestion_server/ingestion_server/indexer.py", line 420, in update
ingestion_server_1  |     self.replicate(model_name, model_name, query)
ingestion_server_1  |   File "/ingestion_server/ingestion_server/indexer.py", line 275, in replicate
ingestion_server_1  |     es_batch = self.pg_chunk_to_es(
ingestion_server_1  |   File "/ingestion_server/ingestion_server/indexer.py", line 443, in pg_chunk_to_es
ingestion_server_1  |     if not (row[schema["removed_from_source"]] or row[schema["deleted"]]):
ingestion_server_1  | KeyError: 'deleted'
```

## Additional context
<!-- Add any other context about the problem here; or delete the section entirely. -->

## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in resolving this bug.



Discussion:

Resolving patch:
diff --git a/ingestion_server/ingestion_server/indexer.py b/ingestion_server/ingestion_server/indexer.py
--- a/ingestion_server/ingestion_server/indexer.py
+++ b/ingestion_server/ingestion_server/indexer.py
@@ -414,9 +414,17 @@ def reindex(self, model_name: str, distributed=None):
 
     def update(self, model_name: str, since_date):
         log.info(f"Updating index {model_name} with changes since {since_date}")
+        deleted, mature = get_existence_queries(model_name)
         query = SQL(
-            "SELECT * FROM {model_name} " "WHERE updated_on >= {since_date};"
-        ).format(model_name=Identifier(model_name), since_date=Literal(since_date))
+            "SELECT *, {deleted}, {mature} "
+            "FROM {model_name} "
+            "WHERE updated_on >= {since_date};"
+        ).format(
+            deleted=deleted,
+            mature=mature,
+            model_name=Identifier(model_name),
+            since_date=Literal(since_date),
+        )
         self.replicate(model_name, model_name, query)
 
     @staticmethod


Test patch:
diff --git a/ingestion_server/test/integration_test.py b/ingestion_server/test/integration_test.py
--- a/ingestion_server/test/integration_test.py
+++ b/ingestion_server/test/integration_test.py
@@ -320,3 +320,41 @@ def test_upstream_indexed_audio(self):
         search_response = es.search(index="audio", body=es_query)
         msg = "There should be 5000 audio tracks in Elasticsearch after ingestion."
         self.assertEquals(search_response["hits"]["total"]["value"], 5000, msg)
+
+    @pytest.mark.order(8)
+    def test_update_index_images(self):
+        """
+        Check that the image data can be updated from the API database into
+        Elasticsearch.
+        """
+        req = {
+            "model": "image",
+            "action": "UPDATE_INDEX",
+            "since_date": "1999-01-01",
+            "callback_url": bottle_url,
+        }
+        res = requests.post(f"{ingestion_server}/task", json=req)
+        stat_msg = "The job should launch successfully and return 202 ACCEPTED."
+        self.assertEqual(res.status_code, 202, msg=stat_msg)
+
+        # Wait for the task to send us a callback.
+        assert self.__class__.cb_queue.get(timeout=120) == "CALLBACK!"
+
+    @pytest.mark.order(9)
+    def test_update_index_audio(self):
+        """
+        Check that the audio data can be updated from the API database into
+        Elasticsearch.
+        """
+        req = {
+            "model": "audio",
+            "action": "UPDATE_INDEX",
+            "since_date": "1999-01-01",
+            "callback_url": bottle_url,
+        }
+        res = requests.post(f"{ingestion_server}/task", json=req)
+        stat_msg = "The job should launch successfully and return 202 ACCEPTED."
+        self.assertEqual(res.status_code, 202, msg=stat_msg)
+
+        # Wait for the task to send us a callback.
+        assert self.__class__.cb_queue.get(timeout=120) == "CALLBACK!"