Repository: openfun/richie

Issue:
"Rare" filter values cannot be removed from search
## Bug Report

**Expected behavior/code**
In addition to the "top facets" (or top filter values) as defined by any given search, all currently selected filter values should be available in the `filters` field of a course search request response.

**Actual Behavior**
Selected filter values that are not in the "top facets" are not included. This makes it impossible for the user to interact with them.
This specific situation can happen if a user starts selecting filter values, picks a "rare" value that is more frequent in the subset of courses filtered by the

**Steps to Reproduce**
- select a bunch of filter values
- select a "rare" filter value that was not available in the default "top facet" but is now available because it happens to be more frequent in the subset of courses your current active filters has selected
- remove the more common filter values
- the "rare" filter value is not visible but is still enabled (see the URL)

(once we add a clean way to directly select "rare" filter values, this will become even more critical)

**Environment**
- Richie version: current master
- Platform: N/A

**Possible Solution**
`FilterDefinitions`' `get_faceted_definition` methods need to explicitly ensure any active value related to their filter is included in the slice of `filters` they are generating.

**Additional context/Screenshots**
Note: this was not an issue before as the client built the filters from complete datasets. Now that the server sends a minimal amount of data (eg #476), it needs to ensure all necessary data is included.


Discussion:

Resolving patch:
diff --git a/src/richie/apps/search/filter_definitions/courses.py b/src/richie/apps/search/filter_definitions/courses.py
--- a/src/richie/apps/search/filter_definitions/courses.py
+++ b/src/richie/apps/search/filter_definitions/courses.py
@@ -44,12 +44,13 @@ def aggs_include(self):
         Returns:
         --------
             string: a regex depending on filters configuration in settings and pages in the CMS:
-                - "^$" if the `reverse_id` does not correspond to any published page which will
-                    will not match any value and return an empty list of facets,
-                - ".*-0001.{4}" if the `reverse_id` points to a published page with path "0001"
-                    this will match ids of the children of this page that will be of the form
-                    P-00010001 if they have children or L-00010001 if they are leafs,
-                ' ".*" if no `reverse_id` is set (delegated to super) which will match all values.
+            - "": the empty string, if the `reverse_id` does not correspond to any published
+                page which will not match any value and return an empty list of facets,
+            - ".*-0001.{4}": if the `reverse_id` points to a published page with path "0001"
+                this will match ids of the children of this page that will be of the form
+                P-00010001 if they have children or L-00010001 if they are leafs,
+            ' ".*": if no `reverse_id` is set (delegated to super) which will
+                match all values.
         """
         if self.reverse_id:
             if self._aggs_include is None:
@@ -58,7 +59,7 @@ def aggs_include(self):
                         publisher_is_draft=False, reverse_id=self.reverse_id
                     )
                 except Page.DoesNotExist:
-                    return "^$"
+                    return ""
                 else:
                     self._aggs_include = ".*-{path:s}.{{{steplen:d}}}".format(
                         path=page.node.path, steplen=page.node.steplen
@@ -124,7 +125,12 @@ def get_faceted_definitions(self, facets, *args, **kwargs):
         """
         Build the filter definition's values from base definition and the faceted keys in the
         current language.
-        Those provide us with the keys and counts that we just have to consume.
+        Those provide us with the keys and counts that we just have to consume. They come from:
+        - a bucket with the top facets in decreasing order of counts,
+        - specific facets for values that were select in the querystring (we must force them
+          because they may not be in the n top facet counts but we must make sure we keep it
+          so that it remains available as an option so the user sees it and can unselect it)
+
         We resort to the `get_i18n_names` method to get the internationalized human names.
         """
         # Convert the keys & counts from ElasticSearch facets to a more readily consumable format
@@ -150,6 +156,16 @@ def get_faceted_definitions(self, facets, *args, **kwargs):
             {},
         )
 
+        # Add the facets that were forced because their value was selected in querystring
+        # Their result is of the form: {'organizations@P-0001': {'doc_count': 12}}
+        key_count_map.update(
+            {
+                key.split("@")[1]: facet["doc_count"]
+                for key, facet in facets.items()
+                if "{:s}@".format(self.name) in key  # 6 times faster than startswith
+            }
+        )
+
         # Get internationalized names for all our keys
         key_i18n_name_map = self.get_i18n_names([key for key in key_count_map])
 
diff --git a/src/richie/apps/search/filter_definitions/mixins.py b/src/richie/apps/search/filter_definitions/mixins.py
--- a/src/richie/apps/search/filter_definitions/mixins.py
+++ b/src/richie/apps/search/filter_definitions/mixins.py
@@ -1,4 +1,5 @@
 """Define mixins to easily compose custom FilterDefinition classes."""
+import re
 
 
 class TermsQueryMixin:
@@ -37,27 +38,57 @@ def get_aggs_fragment(self, queries, data, *args, **kwargs):
         # by the `aggs_include` property:
         include = data[f"{self.name:s}_include"] or self.aggs_include
 
-        return {
+        # Use all the query fragments from the queries *but* the one(s) that filter on the
+        # current filter
+        filter_fragments = [
+            clause
+            for kf_pair in queries
+            for clause in kf_pair["fragment"]
+            if kf_pair["key"] is not self.name
+        ]
+
+        # Terms aggregation will return the n top facet counts among all the values taken
+        # by this field
+        terms_aggs = {
             self.name: {
                 # Rely on the built-in "terms" aggregation to get everything we need
                 "aggregations": {
                     self.name: {"terms": {"field": self.term, "include": include}}
                 },
-                "filter": {
-                    "bool": {
-                        # Use all the query fragments from the queries *but* the one(s) that
-                        # filter on the current filter, as it is handled by ElasticSearch for us
-                        "must": [
-                            clause
-                            for kf_pair in queries
-                            for clause in kf_pair["fragment"]
-                            if kf_pair["key"] is not self.name
-                        ]
-                    }
-                },
+                # Use all the query fragments from the queries *but* the one(s) that
+                # filter on the current filter, as it is handled by ElasticSearch for us
+                "filter": {"bool": {"must": filter_fragments}},
             }
         }
 
+        # Filters aggregation for values that were selected in the querystring (we must force
+        # them because they may not be in the n top facet counts but we must make sure we keep
+        # them so that they remain available as options that the user can see and unselect)
+        terms_aggs.update(
+            {
+                # Create a custom aggregation for each value selected in the querystring
+                # and matching the include regex
+                # eg `organizations@P-0001` & `organizations@P-0002`
+                "{:s}@{:s}".format(self.name, value): {
+                    "filter": {
+                        "bool": {
+                            # Use all the query fragments from the queries *but* the one(s) that
+                            # filter on the current filter: we manually add back the only one that
+                            # is relevant to the current choice.
+                            "must": [{"term": {self.term: value}}]
+                            + filter_fragments
+                        }
+                    }
+                }
+                for value in data.get(self.name, [])
+                # The Elasticsearch include regex matches exact values so we must do the same
+                # by adding ^ (resp. $) at the beginning (resp. at the end) of the pattern.
+                if re.match(f"^{include:s}$", value)
+            }
+        )
+
+        return terms_aggs
+
 
 class ChoicesQueryMixin:
     """A mixin for filter definitions that need to apply predefined queries."""


Test patch:
diff --git a/tests/apps/search/test_filter_definitions.py b/tests/apps/search/test_filter_definitions.py
--- a/tests/apps/search/test_filter_definitions.py
+++ b/tests/apps/search/test_filter_definitions.py
@@ -20,11 +20,11 @@ def test_filter_definitions_indexable_filter_aggs_include_no_page(self):
         """
         for filter_name in ["levels", "subjects", "organizations"]:
             with self.assertNumQueries(1):
-                self.assertEqual(FILTERS[filter_name].aggs_include, "^$")
+                self.assertEqual(FILTERS[filter_name].aggs_include, "")
 
             # The result is not set in cache when a page was not found
             with self.assertNumQueries(1):
-                self.assertEqual(FILTERS[filter_name].aggs_include, "^$")
+                self.assertEqual(FILTERS[filter_name].aggs_include, "")
 
     def test_filter_definitions_indexable_filter_aggs_include_draft_page(self):
         """
@@ -35,11 +35,11 @@ def test_filter_definitions_indexable_filter_aggs_include_draft_page(self):
             CategoryFactory(page_reverse_id=filter_name)
 
             with self.assertNumQueries(1):
-                self.assertEqual(FILTERS[filter_name].aggs_include, "^$")
+                self.assertEqual(FILTERS[filter_name].aggs_include, "")
 
             # The result is not set in cache when a published page was not found
             with self.assertNumQueries(1):
-                self.assertEqual(FILTERS[filter_name].aggs_include, "^$")
+                self.assertEqual(FILTERS[filter_name].aggs_include, "")
 
     def test_filter_definitions_indexable_filter_aggs_include_published_page(self):
         """
diff --git a/tests/apps/search/test_query_courses.py b/tests/apps/search/test_query_courses.py
--- a/tests/apps/search/test_query_courses.py
+++ b/tests/apps/search/test_query_courses.py
@@ -12,7 +12,7 @@
 from elasticsearch.client import IndicesClient
 from elasticsearch.helpers import bulk
 
-from richie.apps.courses.factories import CategoryFactory
+from richie.apps.courses.factories import CategoryFactory, OrganizationFactory
 from richie.apps.search.filter_definitions import FILTERS, IndexableFilterDefinition
 from richie.apps.search.indexers.courses import CoursesIndexer
 
@@ -128,8 +128,8 @@
 )
 class CourseRunsCoursesQueryTestCase(TestCase):
     """
-    Test edge case search queries on underlying course runs to make sure filtering and sorting
-    works as we expect.
+    Test search queries on courses and underlying course runs to make sure filtering and sorting
+    works as expected.
     """
 
     def setUp(self):
@@ -158,8 +158,9 @@ def create_filter_pages():
             - levels page path: 0002
             - organizations page path: 0003
         """
-        for filter_name in ["subjects", "levels", "organizations"]:
-            CategoryFactory(page_reverse_id=filter_name, should_publish=True)
+        CategoryFactory(page_reverse_id="subjects", should_publish=True)
+        CategoryFactory(page_reverse_id="levels", should_publish=True)
+        OrganizationFactory(page_reverse_id="organizations", should_publish=True)
 
     @staticmethod
     def get_expected_courses(courses_definition, course_run_ids):
diff --git a/tests/apps/search/test_query_courses_edge_cases.py b/tests/apps/search/test_query_courses_edge_cases.py
new file mode 100644
--- /dev/null
+++ b/tests/apps/search/test_query_courses_edge_cases.py
@@ -0,0 +1,213 @@
+"""Tests for environment ElasticSearch support."""
+# pylint: disable=too-many-lines
+import json
+import random
+from unittest import mock
+
+from django.conf import settings
+from django.test import TestCase
+
+from elasticsearch.client import IndicesClient
+from elasticsearch.helpers import bulk
+
+from richie.apps.courses.factories import CategoryFactory, OrganizationFactory
+from richie.apps.search.filter_definitions import FILTERS, IndexableFilterDefinition
+from richie.apps.search.indexers.courses import CoursesIndexer
+
+
+@mock.patch.object(  # Avoid having to build the categories and organizations indices
+    IndexableFilterDefinition,
+    "get_i18n_names",
+    side_effect=lambda o: {key: f"#{key:s}" for key in o},
+)
+@mock.patch.object(  # Avoid messing up the development Elasticsearch index
+    CoursesIndexer,
+    "index_name",
+    new_callable=mock.PropertyMock,
+    return_value="test_courses",
+)
+class EdgeCasesCoursesQueryTestCase(TestCase):
+    """
+    Test edge case search queries to make sure facetting works as we expect.
+    """
+
+    def setUp(self):
+        """Reset indexable filters cache before each test so the context is as expected."""
+        super().setUp()
+        self.reset_filter_definitions_cache()
+
+    def tearDown(self):
+        """Reset indexable filters cache after each test to avoid impacting subsequent tests."""
+        super().tearDown()
+        self.reset_filter_definitions_cache()
+
+    @staticmethod
+    def reset_filter_definitions_cache():
+        """Reset indexable filters cache on the `aggs_include` field."""
+        for filter_name in ["levels", "subjects", "organizations"]:
+            # pylint: disable=protected-access
+            FILTERS[filter_name]._aggs_include = None
+
+    @staticmethod
+    def create_filter_pages():
+        """
+        Create pages for each filter based on an indexable. We must create them in the same order
+        as they are instantiated in order to match the node paths we expect:
+            - subjects page path: 0001
+            - levels page path: 0002
+            - organizations page path: 0003
+        """
+        CategoryFactory(page_reverse_id="subjects", should_publish=True)
+        CategoryFactory(page_reverse_id="levels", should_publish=True)
+        OrganizationFactory(page_reverse_id="organizations", should_publish=True)
+
+    def prepare_index(self, courses):
+        """
+        Not a test.
+        This method is doing the heavy lifting for the tests in this class:
+        - prepare the Elasticsearch index,
+        - execute the query.
+        """
+        self.create_filter_pages()
+        # Index these 4 courses in Elasticsearch
+        indices_client = IndicesClient(client=settings.ES_CLIENT)
+        # Delete any existing indexes so we get a clean slate
+        indices_client.delete(index="_all")
+        # Create an index we'll use to test the ES features
+        indices_client.create(index="test_courses")
+        # Use the default courses mapping from the Indexer
+        indices_client.put_mapping(
+            body=CoursesIndexer.mapping, doc_type="course", index="test_courses"
+        )
+        # Add the sorting script
+        settings.ES_CLIENT.put_script(id="state", body=CoursesIndexer.scripts["state"])
+        # Actually insert our courses in the index
+        actions = [
+            {
+                "_id": course["id"],
+                "_index": "test_courses",
+                "_op_type": "create",
+                "_type": "course",
+                **course,
+            }
+            for course in courses
+        ]
+        bulk(
+            actions=actions,
+            chunk_size=settings.ES_CHUNK_SIZE,
+            client=settings.ES_CLIENT,
+        )
+        indices_client.refresh()
+
+    def test_query_courses_rare_facet_force(self, *_):
+        """
+        A facet that is selected in the querystring should always be included in the result's
+        facet counts, even if it is not in the top 10.
+        """
+        organizations = [
+            "L-0003{:04d}".format(o + 1) for o in range(10)
+        ] * 2  # 10 organizations with 2 occurences
+        organizations.append("L-00030011")  # 1 organization with only 1 occurence
+        # => organization with ID "L-00030011" is not in the top ten facets.
+
+        self.prepare_index(
+            [
+                {
+                    "id": index,
+                    "organizations": [id],
+                    "categories": [],
+                    "is_new": False,
+                    "absolute_url": {"en": "url"},
+                    "cover_image": {"en": "image"},
+                    "title": {"en": "title"},
+                    "course_runs": [],
+                }
+                for index, id in enumerate(organizations)
+            ]
+        )
+
+        # If not filter is applied, the 10 top facets should be returned
+        response = self.client.get(f"/api/v1.0/courses/")
+        self.assertEqual(response.status_code, 200)
+        content = json.loads(response.content)
+        self.assertEqual(
+            content["filters"]["organizations"]["values"],
+            [
+                {"count": 2, "human_name": "#L-00030001", "key": "L-00030001"},
+                {"count": 2, "human_name": "#L-00030002", "key": "L-00030002"},
+                {"count": 2, "human_name": "#L-00030003", "key": "L-00030003"},
+                {"count": 2, "human_name": "#L-00030004", "key": "L-00030004"},
+                {"count": 2, "human_name": "#L-00030005", "key": "L-00030005"},
+                {"count": 2, "human_name": "#L-00030006", "key": "L-00030006"},
+                {"count": 2, "human_name": "#L-00030007", "key": "L-00030007"},
+                {"count": 2, "human_name": "#L-00030008", "key": "L-00030008"},
+                {"count": 2, "human_name": "#L-00030009", "key": "L-00030009"},
+                {"count": 2, "human_name": "#L-00030010", "key": "L-00030010"},
+            ],
+        )
+
+        # If organizations are selected, they should be included in the facet counts:
+        # - at its place for the one that is in the top 10
+        # - at the end for the one that is not in the top 10 and should not be included
+        #   if it wasn't forced...
+        response = self.client.get(
+            f"/api/v1.0/courses/?organizations=L-00030002&organizations=L-00030011"
+        )
+        self.assertEqual(response.status_code, 200)
+        content = json.loads(response.content)
+
+        self.assertEqual(
+            content["filters"]["organizations"]["values"],
+            [
+                {"count": 2, "human_name": "#L-00030001", "key": "L-00030001"},
+                {"count": 2, "human_name": "#L-00030002", "key": "L-00030002"},
+                {"count": 2, "human_name": "#L-00030003", "key": "L-00030003"},
+                {"count": 2, "human_name": "#L-00030004", "key": "L-00030004"},
+                {"count": 2, "human_name": "#L-00030005", "key": "L-00030005"},
+                {"count": 2, "human_name": "#L-00030006", "key": "L-00030006"},
+                {"count": 2, "human_name": "#L-00030007", "key": "L-00030007"},
+                {"count": 2, "human_name": "#L-00030008", "key": "L-00030008"},
+                {"count": 2, "human_name": "#L-00030009", "key": "L-00030009"},
+                {"count": 2, "human_name": "#L-00030010", "key": "L-00030010"},
+                {"count": 1, "human_name": "#L-00030011", "key": "L-00030011"},
+            ],
+        )
+
+    def test_query_courses_rare_facet_no_include_match(self, *_):
+        """
+        A facet that is selected in the querystring should not be included in the result's
+        facet counts if it does not match the include regex.
+        """
+        organizations = [
+            "L-0003{:04d}".format(o + 1) for o in range(10)
+        ]  # 10 organizations matching the include regex
+        organizations.append(
+            "L-000300010001"
+        )  # 1 organization not matching the include regex
+
+        self.prepare_index(
+            [
+                {
+                    "id": index,
+                    "organizations": random.sample(
+                        organizations, random.randint(1, len(organizations))
+                    ),
+                    "categories": [],
+                    "is_new": False,
+                    "absolute_url": {"en": "url"},
+                    "cover_image": {"en": "image"},
+                    "title": {"en": "title"},
+                    "course_runs": [],
+                }
+                for index, id in enumerate(organizations)
+            ]
+        )
+
+        response = self.client.get(f"/api/v1.0/courses/?organizations=L-000300010001")
+        self.assertEqual(response.status_code, 200)
+        content = json.loads(response.content)
+
+        self.assertNotIn(
+            "L-000300010001",
+            [o["key"] for o in content["filters"]["organizations"]["values"]],
+        )