Repository: mdn/kuma

Issue:
Create Lightweight Elasticsearch Dev Instance
About 60% of the bytes we transfer from MySQL into Elasticsearch is a field called `html_attributes`. Actually, here's a breakdown:
```
59.7%     html_attributes
35.0%     content
3.5%      css_classnames
0.8%      summary
0.3%      kumascript_macros
0.3%      tags
0.1%      slug
0.1%      modified
0.1%      title
0.0%      id
0.0%      locale
0.0%      boost
0.0%      parent
```

The only way to search on this `html_attributes` (or `css_classnames` for that matter) is if you find this page: https://developer.mozilla.org/en-US/docs/MDN/Contribute/Tools/Search 
but I'm not even sure any of that is working. 
https://developer.mozilla.org/en-US/search?locale=en-US&html_attributes=style will not work and https://wiki.developer.mozilla.org/en-US/search?locale=en-US&html_attributes=style doesn't find anything. 

Having that much data into Elasticsearch matters for developers using the `docker-compose` tooling. If you have over 5,000 documents indexed in Elasticsearch need to massively increase how much RAM you're giving to docker. 

It would be nice to not have to index that stuff for your local dev instance. 


Discussion:
A simple solution would be something like this:

```python
# in settings/common.py
INDEX_HTML_ATTRIBUTES = not DEBUG
INDEX_CSS_CLASSNAMES = not DEBUG

# in kuma/wiki/search.py
'html_attributes': cls.case_insensitive_keywords(
                obj.extract.html_attributes()) if settings.INDEX_HTML_ATTRIBUTES else [],
```


For the record, this is what the doc can look like that gets sent from `kuma/wiki/search.py` over to ES: https://gist.github.com/peterbe/99fce14a96d51bb0728563acb3d1756a

Resolving patch:
diff --git a/kuma/settings/common.py b/kuma/settings/common.py
--- a/kuma/settings/common.py
+++ b/kuma/settings/common.py
@@ -1820,6 +1820,19 @@ def get_user_url(user):
 # One good example is: https://github.com/ubernostrum/django-soapbox/issues/5
 MIGRATION_MODULES = {"soapbox": "kuma.soap_migrations"}
 
+# html_attributes and css_classnames get indexed into Elasticsearch on every
+# document when sent in. These can be very memory consuming since the
+# 'html_attributes' makes up about 60% of the total weight.
+# Refer to this GitHub issue for an estimate of their weight contribution:
+# https://github.com/mdn/kuma/issues/6264#issue-539922604
+# Note that the only way to actually search on these fields is with a manual
+# use of the search v1 API. There is no UI at all for searching on something
+# in the 'html_attributes' or the 'css_classnames'.
+# By disabling indexing of these, in your local dev environment, your local
+# Elasticsearch instance will be a LOT smaller.
+INDEX_HTML_ATTRIBUTES = config("INDEX_HTML_ATTRIBUTES", cast=bool, default=not DEBUG)
+INDEX_CSS_CLASSNAMES = config("INDEX_CSS_CLASSNAMES", cast=bool, default=not DEBUG)
+
 # For local development you might want to set this to a hostname provided to
 # you by a tunneling service such as ngrok.
 STRIPE_WEBHOOK_HOSTNAME = config("STRIPE_WEBHOOK_HOSTNAME", default=None)
diff --git a/kuma/wiki/search.py b/kuma/wiki/search.py
--- a/kuma/wiki/search.py
+++ b/kuma/wiki/search.py
@@ -5,7 +5,6 @@
 from django.conf import settings
 from django.db.models import Q
 from django.utils.html import strip_tags
-from django.utils.translation import gettext_lazy as _
 from elasticsearch.helpers import bulk
 from elasticsearch_dsl import document, field
 from elasticsearch_dsl.connections import connections
@@ -87,10 +86,14 @@ def from_django(cls, obj):
             ),
             "css_classnames": cls.case_insensitive_keywords(
                 obj.extract.css_classnames()
-            ),
+            )
+            if settings.INDEX_CSS_CLASSNAMES
+            else None,
             "html_attributes": cls.case_insensitive_keywords(
                 obj.extract.html_attributes()
-            ),
+            )
+            if settings.INDEX_HTML_ATTRIBUTES
+            else None,
         }
 
         if obj.parent:
@@ -298,14 +301,8 @@ def reindex_all(cls, chunk_size=500, index=None, percent=100):
             ]
             chord_flow(pre_task, index_tasks, post_task).apply_async()
 
-        message = _(
-            "Indexing %(total)d documents into %(total_chunks)d chunks of "
-            "size %(size)d into index %(index)s."
-            % {
-                "total": total,
-                "total_chunks": total_chunks,
-                "size": chunk_size,
-                "index": index.prefixed_name,
-            }
+        message = (
+            f"Indexing {total:,} documents into {total_chunks} chunks of "
+            f"size {chunk_size} into index {index.prefixed_name}."
         )
         return message


Test patch:
diff --git a/kuma/settings/testing.py b/kuma/settings/testing.py
--- a/kuma/settings/testing.py
+++ b/kuma/settings/testing.py
@@ -96,6 +96,11 @@
 STRIPE_SECRET_KEY = "testing"
 STRIPE_PLAN_ID = "testing"
 
+# For legacy reasons, the tests assume these are always true so don't
+# let local overrides take effect.
+INDEX_HTML_ATTRIBUTES = True
+INDEX_CSS_CLASSNAMES = True
+
 # Amount for the monthly subscription.
 # It's hardcoded here in case some test depends on the number and it futureproofs
 # our tests to not deviate when the actual number changes since that number