Repository: WordPress/openverse-api

Issue:
Filtering by extension for audio doesn't work as expected
## Problem
<!-- Describe a problem solved by this feature; or delete the section entirely. -->
When you filter audio search by extension, you don't get results that you would expect. Only `mp3` category returns any results.

## Description
<!-- Describe the feature and how it solves the problem. -->
To get the audio extension field in ElasticSearch, we use the URL's last part.
https://github.com/WordPress/openverse-api/blob/5c9b654a9e3ab55932de42aa76ae2c17f5e85d1d/ingestion_server/ingestion_server/elasticsearch_models.py#L136-L144

This doesn't work well for audio because many URLs don't have an extension at all. Fortunately, most of the audio files should have the `filetype` attribute saved in the database, so we could use that as the `extension` value.

Marking this as high priority because having a category filter in the UI that is not working properly is confusing for the users.

## Alternatives
<!-- Describe any alternative solutions or features you have considered. How is this feature better? -->
Not really alternative, but additional solutions would be to use the extensions of the `alt_files` in addition to the main file extension.

## Additional context
<!-- Add any other context about the feature here; or delete the section entirely. -->

## Implementation
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in implementing this feature.



Discussion:
@obulat This makes me feel doubly that we should be mapping file extensions on ingestion, in the catalog, rather than trying to derive it in the API and/or the frontend. Otherwise our ES logic will become massively complicated, trying to map individual provider quirks for showing the file format per result...

During ingestion we could at least make a HEAD request for the providers that we don't know for sure what the file type is and use the content-type header as a mime-type maybe?

Anyway, it seems like the place to normalize data is in ingestion rather than further down the data pipeline (where it will probably get duplicated between API and frontend).
I agree that we should get the filetype on ingestion.
However, because we have `alt_files`, we also need to decide whether we want to add some logic in the ES, too. Or we can say that we only use the main file's filetype (extension), and do not include the items with the selected extension only in the `alt_files`.
For example, if the file is `galah.ogg`, but has `galah.mp3` in the `alt_files`, do we show it when we search for `mp3` 'galah' files? This would require some ES logic, I suppose.
I think we'd _only_ filter on the downloadable file types, right? So basically only filter on the file types of `alt_files`. Otherwise we're misleading someone in the example you gave if we filter it as an ogg file but they can only download an mp3.

We don't currently offer the preview audio as a download option (unless there are no alt_files OR if that file is already included in alt files). This is because some providers require logging in to get the proper audio download and we wanted to respect that, so we partition the preview audio as a totally separate thing.

Ideally we'd never have an empty alt_files for any provider (and probably should rename `alt_files` to like `downloadable_files` or something clearer, they're not "alternatives" in the case for audio, they are the actual downloadable files and the preview is "just" a preview, for which quality and file type information should be completely ignored from a consumer perspective).

At least that's my understanding of it. The audio alt files stuff has been such a pain and it feels like we could do with taking a step back and thinking of it holistically, from the top to the bottom of the stack to make sure the usage and data that we're collecting is clear and fulfills all the requirements necessary. We keep coming back to confusions about the alt files and how they work and how we're meant to work with them, it seems clear to me that we've missed something in defining the requirements for this.
cc @WordPress/openverse-developers because this conversation is a lot bigger than just me and Olga :slightly_smiling_face: 
Yeah, the name `alt_files` has likely been a source of the confusion. `downloadable_files` or simply `files` would be much clearer. And agreed, the format of the _preview_ shouldn't be considered in the filtering. 

> We don't currently offer the preview audio as a download option (unless there are no alt_files OR if that file is already included in alt files). This is because some providers require logging in to get the proper audio download and we wanted to respect that, so we partition the preview audio as a totally separate thing.

It might help to _only_ use `alt_files` for the download options in the frontend, and add the preview audio to `alt_files` at the Catalog level if there are no `alt_files`. Then we don't have to fiddle with as much on the frontend: The download button simply lists all `alt_files` instead of building it's own list with the preview file. 
The main reason for the differentiation between the main file and `alt_files` was that the main file is supposed to be the best quality while the `alt_files` are other lower quality/smaller sizes. This is similar to [how Wikimedia Commons shows "Original file" and "Other resolutions"](https://commons.wikimedia.org/wiki/File:Pacific_Ocean_laea_location_map.svg#mw-jump-to-license).
> The main reason for the differentiation between the main file and `alt_files` was that the main file is supposed to be the best quality while the `alt_files` are other lower quality/smaller sizes. This is similar to [how Wikimedia Commons shows "Original file" and "Other resolutions"](https://commons.wikimedia.org/wiki/File:Pacific_Ocean_laea_location_map.svg#mw-jump-to-license).

That's super helpful context, thank you! I think our use case ultimately ends up being different from Wikimedia Commons, since the first file folks encounter is the preview rather than the original file. To sum up the discussion here, I think the following might be the best course of action:
* Rename `alt_files` to `downloadable_files`
* Add steps in the provider scripts that:
  * Capture extension information for each of the `downloadable_files`
  * Add the preview file information to `downloadable_files` iff no other `downloadable_files` exist
* Add the extensions as a list to the audio Elasticsearch model so they can be used in searches

Does that capture everything? Anything I'm missing or anything else we should consider?
+1 to your point about our usage being different than WM Commons. We don't control the full stack of information or policies that all providers give us and our users access to so our data model needs to be more flexible/specific than WM's.

> Rename alt_files to downloadable_files

Note that this will require a new API version as it is a non-backwards compatible breaking change. Unfortunately in a versioned API you just have to live with bad names forever until you are ready to cut a new version (or else break the implicit contract of the version prefix and lose trust from consumers of the API). `alt_files` as a property name on the API response is the smallest problem here so personally I'm fine leaving out changing it until we're ready to work on cutting a new API version. Solving the problems with the _data_ in it (the other part of your suggested approach) is more important.

We could send both alt_files and downloadable_files, but then we're really making a bigger API response for little gain.

The rest sounds great though, Madison.
Ah jeez, I forgot about API versioning! :confounded: We'd also need to consider backfills, particularly for Wikimedia. Though I don't think we're losing out on any _current_ functionality if we just rename the field, so there may not be as much urgency there.
Yeah I'm fine with preserving the user-facing `alt_files` name. I wonder how we could add documentation like 'this will be renamed to {x} in a future release. Could we mark the current field as deprecated with a note, or something?
I don't think it's necessary to pre-emptively document the differences between current and future (still theoretical) API versions. The field name will never change in the version so for people who are not interested in new versions of the API it doesn't matter. For everyone else (who are keeping up with changelogs) they will be notified when it's available and when they upgrade to the new endpoint version.
There was certainly a confusion somewhere along the way in that a difference between preview files and downloadable files was assumed to be the foundation of the audio model, but actually, the audio table was conceived as @dhruvkb  mentioned:

> the main file is supposed to be the best quality while the alt_files are other lower quality/smaller sizes

The main audio is in most cases a full audio track, not a preview. If I recall correctly, we were only using previews as a workaround in the case of Freesound, to be able to reproduce something in Openverse, because of its API limitation that requires OAuth authentication to access a full audio track. We don't store any `alt_files` from Wikimedia or Jamendo either. So I believe we shouldn't base the model on whether the audio can be downloaded or not, since it's even possible that we remove the download button in the future, in this way the model can be more flexible.

That said I agree that we should get extension/file type, and actually _ALL_ the info needed on ingestion, including the waveform since we are here.

Here's a snapshot of our data:
![image](https://user-images.githubusercontent.com/10214785/163046828-636448f5-cfae-45b4-881c-cf9ee3ee13e5.png)

Summarizing [some discussion from our public Slack](https://wordpress.slack.com/archives/C02012JB00N/p1649776492982129):
1. We'd still like to rename `alt_files` to `downloadable_files`, as the name feels more appropriate for our intended usecase.
2. In cases where `downloadable_files` is not null, the `url` field semantically represents the **preview URL** and may not actually correspond to a downloadable file.
3. The extension logic would be as follows:
    1. If a record has no `downloadable_files`, the `url` and `filetype` fields _on the record_ represent the downloadable file
    1. If a record has `downloadable_files`, only the entries within the `downloadable_files` array represent downloadable files

The question we have open is whether we want that logic encapsulated in code, or removed entirely by populating the `downloadable_files` array with the existing record info if it is empty (as described above). Duplicating the data within the record would certainly be easier, but I'm concerned at how a size increase would affect indexing & speed. It's possible that we could add this logic into the media type views - we only keep one copy of the data, but when the view is materialized it performs some JSON manipulation to add any missing info to `downloadable_files`. The view approach would still result in some duplicated data in the API once the view is copied from the catalog during the data refresh.

It may also be straightforward to extract the information we need in the Elasticsearch models. I suspect that would be pretty easy, but I'd need to take a look at the codebase. That to say, I'll attempt the ES path while also trying to get a better understanding of how data duplication might affect our already large database.

> We'd still like to rename alt_files to downloadable_files, as the name feels more appropriate for our intended usecase.

Just want to re-iterate, we can rename this internally but the external name for this field will have to remain `alt_files` as long as we're still on the v1 of the audio api endpoint.

> In cases where downloadable_files is not null

Kind of a nit, but just want to be sure to cover the edge case, downloadable_files could also be _empty_ otherwise (length == 0). I don't know if we have actual data that looks like that though.

The plan sounds good though! Thank you for summarizing the discussion from slack.
@AetherUnbound Are you actively working on this? It's a perfect candidate to move to the "In progress" column. The resolution is required to move forward in the Audio milestone.
I am not actively on it @krysal, please feel free to take it if you have time!

Resolving patch:
diff --git a/api/catalog/api/examples/audio_responses.py b/api/catalog/api/examples/audio_responses.py
--- a/api/catalog/api/examples/audio_responses.py
+++ b/api/catalog/api/examples/audio_responses.py
@@ -70,7 +70,6 @@
                 "audio_set": None,
                 "attribution": None,
                 "filesize": None,
-                "filetype": None,
             },
         ],
     },
@@ -86,6 +85,13 @@
 
 audio_stats_200_example = {
     "application/json": [
+        {
+            "source_name": "freesound",
+            "display_name": "Freesound",
+            "source_url": "https://freesound.org/",
+            "logo_url": None,
+            "media_count": 828,
+        },
         {
             "source_name": "jamendo",
             "display_name": "Jamendo",
@@ -98,7 +104,7 @@
             "display_name": "Wikimedia",
             "source_url": "https://commons.wikimedia.org",
             "logo_url": None,
-            "media_count": 4820,
+            "media_count": 3992,
         },
     ]
 }
diff --git a/ingestion_server/ingestion_server/elasticsearch_models.py b/ingestion_server/ingestion_server/elasticsearch_models.py
--- a/ingestion_server/ingestion_server/elasticsearch_models.py
+++ b/ingestion_server/ingestion_server/elasticsearch_models.py
@@ -107,7 +107,6 @@ def get_instance_attrs(row, schema):
             "creator": row[schema["creator"]],
             "creator_url": row[schema["creator_url"]],
             "url": row[schema["url"]],
-            "extension": Media.get_extension(row[schema["url"]]),
             "license": row[schema["license"]].lower(),
             "license_version": row[schema["license_version"]],
             "license_url": Media.get_license_url(meta),
@@ -132,17 +131,6 @@ def parse_description(metadata_field):
         except TypeError:
             return None
 
-    @staticmethod
-    def get_extension(url):
-        """
-        Get the extension from the last segment of the URL separated by a dot.
-        """
-        extension = url.split(".")[-1].lower()
-        if "/" in extension or extension is None:
-            return None
-        else:
-            return extension
-
     @staticmethod
     def get_license_url(meta_data):
         """
@@ -261,6 +249,7 @@ def database_row_to_elasticsearch_doc(row, schema):
             thumbnail=row[schema["thumbnail"]],
             category=category,
             aspect_ratio=aspect_ratio,
+            extension=extension,
             size=size,
             authority_boost=authority_boost,
             max_boost=max(popularity or 1, authority_boost or 1),
@@ -280,6 +269,19 @@ def get_aspect_ratio(height, width):
             aspect_ratio = Image.AspectRatios.SQUARE.name
         return aspect_ratio.lower()
 
+    @staticmethod
+    def get_extension(url):
+        """
+        Get the extension from the last segment of the URL separated by a dot.
+        TODO: Use the `filetype` field once the following issue is completed:
+        https://github.com/WordPress/openverse-catalog/issues/536
+        """
+        extension = url.split(".")[-1].lower()
+        if not extension or "/" in extension:
+            return None
+        else:
+            return extension
+
     @staticmethod
     def get_size(height, width):
         if height is None or width is None:
@@ -314,6 +316,10 @@ class Index:
 
     @staticmethod
     def database_row_to_elasticsearch_doc(row, schema):
+        alt_files = row[schema["alt_files"]]
+        filetype = row[schema["filetype"]]
+        extension = Audio.get_extensions(filetype, alt_files)
+
         meta = row[schema["meta_data"]]
         provider = row[schema["provider"]]
         authority_boost = Audio.get_authority_boost(meta, provider)
@@ -330,12 +336,21 @@ def database_row_to_elasticsearch_doc(row, schema):
             category=row[schema["category"]],
             duration=row[schema["duration"]],
             length=length,
+            filetype=filetype,
+            extension=extension,
             authority_boost=authority_boost,
             max_boost=max(popularity or 1, authority_boost or 1),
             min_boost=min(popularity or 1, authority_boost or 1),
             **attrs,
         )
 
+    @staticmethod
+    def get_extensions(filetype, alt_files):
+        if not alt_files:
+            return filetype
+
+        return [file["filetype"] for file in alt_files]
+
     @staticmethod
     def get_length(duration):
         if not duration:
diff --git a/ingestion_server/ingestion_server/es_mapping.py b/ingestion_server/ingestion_server/es_mapping.py
--- a/ingestion_server/ingestion_server/es_mapping.py
+++ b/ingestion_server/ingestion_server/es_mapping.py
@@ -101,6 +101,7 @@ def index_settings(table_name):
                 "fields": {"keyword": {"ignore_above": 256, "type": "keyword"}},
                 "type": "text",
             },
+            "filetype": {"type": "keyword"},
             "created_on": {"type": "date"},
             "tags": {
                 "properties": {


Test patch:
diff --git a/api/test/audio_integration_test.py b/api/test/audio_integration_test.py
--- a/api/test/audio_integration_test.py
+++ b/api/test/audio_integration_test.py
@@ -49,7 +49,7 @@ def test_search_category_filtering_fails(audio_fixture):
 
 
 def test_search_all_excluded():
-    search_all_excluded("audio", ["jamendo", "wikimedia_audio"])
+    search_all_excluded("audio", ["freesound", "jamendo", "wikimedia_audio"])
 
 
 def test_search_source_and_excluded():
@@ -82,7 +82,7 @@ def test_audio_thumb(audio_fixture):
 
 
 def test_audio_detail_without_thumb():
-    resp = requests.get(f"{API_URL}/v1/audio/00289ffb-0c74-4008-8388-0bf6d8173dee")
+    resp = requests.get(f"{API_URL}/v1/audio/44540200-91eb-483d-9e99-38ce86a52fb6")
     assert resp.status_code == 200
     parsed = json.loads(resp.text)
     assert parsed["thumbnail"] is None
diff --git a/ingestion_server/test/unit_tests/conftest.py b/ingestion_server/test/unit_tests/conftest.py
--- a/ingestion_server/test/unit_tests/conftest.py
+++ b/ingestion_server/test/unit_tests/conftest.py
@@ -1,7 +1,66 @@
 import datetime
 from uuid import uuid4
 
-from ingestion_server.elasticsearch_models import Image
+from ingestion_server.elasticsearch_models import Audio, Image
+
+
+def create_mock_audio(override=None):
+    """
+    Produce a mock audio. Override default fields by passing in a dict with the
+    desired keys and values.
+
+    For example, to make an image with a custom title and default everything
+    else:
+    >>> create_mock_audio({'title': 'My title'})
+    :return:
+    """
+    test_popularity = {"downloads": 3}
+    license_url = "http://creativecommons.org/publicdomain/zero/1.0/"
+    meta_data = {"popularity_metrics": test_popularity, "license_url": license_url}
+    test_data = {
+        "id": 0,
+        "title": "Torneira de água",
+        "identifier": str(uuid4()),
+        "creator": "MartaSarmento",
+        "creator_url": "https://freesound.org/people/MartaSarmento",
+        "tags": [{"name": "test", "accuracy": 0.9}],
+        "created_on": datetime.datetime.now(),
+        "foreign_landing_url": "https://freesound.org/people/MartaSarmento/sounds/509257",
+        "url": "https://cdn.freesound.org/previews/509/509257_11129467-hq.mp3",
+        "genres": ["genre1", "genre2"],
+        "category": "test_category",
+        "duration": 8544,
+        "bit_rate": 128000,
+        "sample_rate": None,
+        "filesize": 168919,
+        "filetype": "mp3",
+        "provider": "test",
+        "source": "test",
+        "license": "cc0",
+        "license_version": "1.0",
+        "mature": False,
+        "meta_data": meta_data,
+        "alt_files": [
+            {
+                "url": "https://freesound.org/apiv2/sounds/509257/download/",
+                "bit_rate": "70000",
+                "filesize": "75591",
+                "filetype": "m4a",
+                "sample_rate": "44100",
+            }
+        ],
+    }
+    if override:
+        for k, v in override.items():
+            test_data[k] = v
+    schema = {}
+    row = []
+    idx = 0
+    for k, v in test_data.items():
+        schema[k] = idx
+        row.append(v)
+        idx += 1
+    return Audio.database_row_to_elasticsearch_doc(row, schema)
 
 
 def create_mock_image(override=None):
diff --git a/ingestion_server/test/unit_tests/test_es_audio_model.py b/ingestion_server/test/unit_tests/test_es_audio_model.py
new file mode 100644
--- /dev/null
+++ b/ingestion_server/test/unit_tests/test_es_audio_model.py
@@ -0,0 +1,10 @@
+from test.unit_tests.conftest import create_mock_audio
+
+
+class TestAudio:
+    @staticmethod
+    def test_extension():
+        single_file = create_mock_audio({"alt_files": None})
+        assert single_file.extension == "mp3"
+        alt_files = create_mock_audio()
+        assert alt_files.extension == ["m4a"]
diff --git a/ingestion_server/test/unit_tests/test_es_models.py b/ingestion_server/test/unit_tests/test_es_image_model.py
similarity index 100%
rename from ingestion_server/test/unit_tests/test_es_models.py
rename to ingestion_server/test/unit_tests/test_es_image_model.py