Repository: WordPress/openverse-api

Issue:
Type errors in the ingestion server 'cleanup' step
## Description
<!-- Concisely describe the bug. -->

Recently in production, the cleanup steps on the ingestion server `INGEST_UPSTREAM` task failed with the following error output:

```shell
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/ingestion_server/ingestion_server/tasks.py", line 106, in run
    reload_upstream(self.model)
  File "/ingestion_server/ingestion_server/ingest.py", line 293, in reload_upstream
    clean_image_data(table)
  File "/ingestion_server/ingestion_server/cleanup.py", line 290, in clean_image_data
    pool.starmap(_clean_data_worker, jobs)
  File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
TypeError: '<' not supported between instances of 'str' and 'float'
```

This means that when comparing tags, tags with an "accuracy" property are somehow strings, and fail when attempting to compare them to our 	TAG_MIN_CONFIDENCE` constant.

https://github.com/WordPress/openverse-api/blob/3768ebced5d70f919e595e227ca0dd02c19f5cf2/ingestion_server/ingestion_server/cleanup.py#L106-L106

We should probably only make the comparison if the tag accruracy is a `float`, or otherwise attempt to convert the string.


I'd like to see an example of which rows have these `accuracy` fields as strings, and try to fix them earlier in the pipeline, but it seems quite a difficult or computationally expensive thing to query for.

Ah, I found an example, from the common crawl data. It seems reasonable to convert these into floats rather than skip them.

```
[{"name": "home", "accuracy": "0.98252", "provider": "clarifai"}, {"name": "road", "accuracy": "0.97564", "provider": "clarifai"}, {"name": "house", "accuracy": "0.9734", "provider": "clarifai"}, {"name": "building", "accuracy": "0.9616", "provider": "clarifai"}, {"name": "tree", "accuracy": "0.96117", "provider": "clarifai"}, {"name": "landscape", "accuracy": "0.9558", "provider": "clarifai"}, {"name": "architecture", "accuracy": "0.94331", "provider": "clarifai"}, {"name": "wood", "accuracy": "0.93565", "provider": "clarifai"}, {"name": "travel", "accuracy": "0.9196", "provider": "clarifai"}, {"name": "farm", "accuracy": "0.90478", "provider": "clarifai"}, {"name": "bungalow", "accuracy": "0.90183", "provider": "clarifai"}, {"name": "guidance", "accuracy": "0.90176", "provider": "clarifai"}, {"name": "daylight", "accuracy": "0.89773", "provider": "clarifai"}, {"name": "fence", "accuracy": "0.88351", "provider": "clarifai"}, {"name": "environment", "accuracy": "0.87694", "provider": "clarifai"}, {"name": "outdoors", "accuracy": "0.8662", "provider": "clarifai"}, {"name": "street", "accuracy": "0.86572", "provider": "clarifai"}, {"name": "sky", "accuracy": "0.85402", "provider": "clarifai"}, {"name": "barn", "accuracy": "0.85313", "provider": "clarifai"}]
```
## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [x] 🙋 I would be interested in resolving this bug.



Discussion:

Resolving patch:
diff --git a/ingestion_server/ingestion_server/cleanup.py b/ingestion_server/ingestion_server/cleanup.py
--- a/ingestion_server/ingestion_server/cleanup.py
+++ b/ingestion_server/ingestion_server/cleanup.py
@@ -104,9 +104,9 @@ def cleanup_tags(tags):
             return None
         for tag in tags:
             below_threshold = False
-            if "accuracy" in tag and tag["accuracy"] < TAG_MIN_CONFIDENCE:
+            if "accuracy" in tag and float(tag["accuracy"]) < TAG_MIN_CONFIDENCE:
                 below_threshold = True
-            if "name" in tag:
+            if "name" in tag and isinstance(tag["name"], str):
                 lower_tag = tag["name"].lower()
                 should_filter = _tag_denylisted(lower_tag) or below_threshold
             else:


Test patch: