Repository: WordPress/openverse-api

Issue:
Add individual command logging for index/constraint application step
## Description
<!-- Describe the feature and how it solves the problem. -->
As part of the investigation around the API response time jumping dramatically during a data refresh, we identified that the index & constraint application step specifically was causing the API degradation. I suspect that there may be a single index/constraint application that is causing the slowness, rather than the entire set of `ALTER`s. In order to test this hypothesis, we need accurate logging for when each step is run.

Specifically, these two query executions should be split up into the individual alters, and each one should be logged when it is run:

https://github.com/WordPress/openverse-api/blob/5435e0e25d13c4b8b4e99ccaa7548a1e71f97683/ingestion_server/ingestion_server/ingest.py#L352

https://github.com/WordPress/openverse-api/blob/5435e0e25d13c4b8b4e99ccaa7548a1e71f97683/ingestion_server/ingestion_server/ingest.py#L359


## Additional context
<!-- Add any other context about the feature here; or delete the section entirely. -->
Similar logging was added for the go-live step in #415

## Implementation
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in implementing this feature.



Discussion:
> Specifically, these two query executions should be split up into the individual alters, and each one should be logged when it is run:

Please log when it is run and when it finishes, maybe even log the delta if possible just so it's nice and easy to diagnose if there's an issue there :slightly_smiling_face: 
Ah, they're run immediately after each other, so I was thinking the logs in CloudWatch would end up looking something like this:

```
2022-05-27 05:23:00,000 INFO ingest.py:356 - ALTER <column> ADD INDEX <foo>
2022-05-27 05:23:10,000 INFO ingest.py:356 - ALTER <column> ADD INDEX <bar>
2022-05-27 05:23:20,000 INFO ingest.py:356 - ALTER <column> ADD INDEX <baz>
2022-05-27 05:23:30,000 INFO ingest.py:356 - ALTER <column> ADD CONSTRAINT <zip>
```

Do you think that'd be sufficient? That would give us the exact indications of when each one starts, and since these are run serially the end can be assumed to just before the next line starts.

I guess for all except the last one?
These are the only steps run after before another log line appears:
https://github.com/WordPress/openverse-api/blob/5435e0e25d13c4b8b4e99ccaa7548a1e71f97683/ingestion_server/ingestion_server/ingest.py#L353-L356

`_update_progress` just sets a value:
https://github.com/WordPress/openverse-api/blob/5435e0e25d13c4b8b4e99ccaa7548a1e71f97683/ingestion_server/ingestion_server/ingest.py#L247-L249

I feel like that gives us a good enough time bounding. If anything though, I can swap the log line and the `_update_progress` one.

Resolving patch:
diff --git a/ingestion_server/ingestion_server/ingest.py b/ingestion_server/ingestion_server/ingest.py
--- a/ingestion_server/ingestion_server/ingest.py
+++ b/ingestion_server/ingestion_server/ingest.py
@@ -138,7 +138,7 @@ def _is_foreign_key(_statement, table):
     return f"REFERENCES {table}(" in _statement
 
 
-def _remap_constraint(name, con_table, fk_statement, table):
+def _remap_constraint(name, con_table, fk_statement, table) -> list[SQL]:
     """Produce ALTER TABLE ... statements for each constraint."""
     alterations = [
         SQL("ALTER TABLE {con_table} DROP CONSTRAINT {name}").format(
@@ -200,7 +200,7 @@ def _generate_delete_orphans(fk_statement, fk_table):
     return del_orphans
 
 
-def _generate_constraints(conn, table: str):
+def _generate_constraints(conn, table: str) -> list[SQL]:
     """
     Using the existing table as a template, generate ALTER TABLE ADD CONSTRAINT
     statements pointing to the new table.
@@ -349,19 +349,23 @@ def reload_upstream(
         create_indices, index_mapping = _generate_indices(downstream_db, table)
         _update_progress(progress, 50.0)
         if create_indices != "":
-            downstream_cur.execute(";\n".join(create_indices))
+            for create_index in create_indices:
+                log.info(f"Running: {create_index}")
+                downstream_cur.execute(create_index)
+        log.info("Done creating indices! Remapping constraints...")
         _update_progress(progress, 70.0)
 
         # Step 7: Recreate constraints from the original table
-        log.info("Done creating indices! Remapping constraints...")
-        remap_constraints = SQL(";\n").join(_generate_constraints(downstream_db, table))
-        if len(remap_constraints.seq) != 0:
-            downstream_cur.execute(remap_constraints)
+        remap_constraints = _generate_constraints(downstream_db, table)
+        if len(remap_constraints):
+            for remap_constraint in remap_constraints:
+                log.info(f"Running: {remap_constraint.as_string(downstream_cur)}")
+                downstream_cur.execute(remap_constraint)
+        log.info("Done remapping constraints! Going live with new table...")
         _update_progress(progress, 99.0)
         slack.verbose(f"`{table}`: Indices & constraints applied | _Next: go-live_")
 
         # Step 8: Promote the temporary table and delete the original
-        log.info("Done remapping constraints! Going live with new table...")
         go_live = get_go_live_query(table, index_mapping)
         log.info(f"Running go-live: \n{go_live.as_string(downstream_cur)}")
         downstream_cur.execute(go_live)


Test patch:
diff --git a/ingestion_server/test/integration_test.py b/ingestion_server/test/integration_test.py
--- a/ingestion_server/test/integration_test.py
+++ b/ingestion_server/test/integration_test.py
@@ -296,7 +296,7 @@ def tearDownClass(cls) -> None:
         log_output = compose_path.parent / "ingestion_logs.txt"
         with log_output.open("w") as file:
             subprocess.run(
-                ["docker-compose", "-f", compose_path.name, "logs"],
+                ["docker-compose", "-f", compose_path.name, "logs", "--no-color"],
                 cwd=compose_path.parent,
                 check=True,
                 stderr=subprocess.STDOUT,