Repository: mdn/kuma

Issue:
Expose the "raw Kuma HTML" in the Document JSON API
The Document JSON API only contains some meta data, the `tocHTML`, the `quickLinksHTML`, and the `bodyHTML`. If you want to find out, to "reverse engineer", **how KS macros were called** you have to  `GET {FULL_DOCUMENT_URL}?raw`. This won't scale if you need to try to convert thousands of documents to static files to put pushed through the static site generation stuff in Stumptown for example. 




Discussion:
I propose we add a new field like `kuma_html` to `kuma.api.v1.views.document_api_data`'s response. 

If we had that, since those are published to an S3 bucket we'd be able to do an S3 batch operation to download N documents and extract the rendered HTML *and* figure out what the HTML became AND what various arguments were when calling `{{Compat(.*?)}}`.
@peterbe Would it be at least equally helpful (maybe even more helpful) to include a list of all of the macro calls and their arguments, rather than the entire raw document? I'm wondering if it might be worthwhile to explore that path in order to reduce the amount of duplication in the data. Or maybe for reverse-engineering purposes it's really helpful to have the macros "in situ"?
All excellent points @escattone and I think it's worth slowing down to carefully test what's needed and what works. I already implemented what this issue says, in a branch, and I now have an S3 bucket full of JSON documents that have the rendered HTML and the raw HTML so with that I *should* be able to produce an auxiliary file structure that I can send through the stumptown pipes. 

This is my skunkworks project and it's nowhere near going into `master` :)

But having this issue will help as a dumping ground for ideas and notes. 
FWIW I've been doing something similar, to scrape Wiki pages for stumptown. For this, most macro calls I will simply remove (by taking the "?raw&macros" and converting to Markdown). But there are some - only `Compat`, `EmbedLiveSample`, `EmbedInteractiveSample` at the moment - that I want to convert into something stumptowny.

And for that I've been using "?raw" to get the version with the macro calls in, then trying to extract macro calls and arguments out of that, then using "?raw&macros" as the source to scrape. So for this (which I think is close to what Peter is doing), yes, definitely, getting direct access to KS calls would be helpful.

But I suppose exactly what we want here depends on exactly what we are trying to accomplish :).


I've been thinking about it some more. But probably not enough. 

Just getting the docs down to disk is hard. I think the best idea is to write a program that more or less gets all the Document JSON API docs down to disk is a task on its own. Then, I think the `scrape-mdn.js` would need to be broken up. Instead of getting one URL at a time, it would get one local file at a time. Then, when running the stuff that `scrape-mdn.js` does it'll read slightly differently but it would also not attempt to say the output is perfect for inclusion into the stumptown-content git repo. Perhaps, next to the `content` directory there'd be a `legacy-content` directory which would look very similar. Next, we'd run `build-json` on *both* `content` and `legacy-content` with a hint to "prefer" the URLs from `content` if there's an overlap on the `mdn_url`. 

I.e. I'll worry about a way to get all the files onto disk from S3. Then I'll modify `scrape-mdn.js` to do what it does but read from disk and lastly output its results in a slightly different location. 

The idea is to do what Will does manually, except we do it in a super automated way, at large scale, and we don't check it into git when we're done instead we just pipe it to `build-json` or `stumptown-renderer` and upload the produced file back into some S3 bucket or something. All with the ability to do it all over again at the press of a button. 

`</loose-thoughts>`
Couldn't we just get a slave db and run a second Kuma in maintenance mode, then we could just use a celery task to write all documents to an S3 bundle?

If this doesn't sound totally stupid, I can expose my idea in more details.

Resolving patch:
diff --git a/kuma/api/tasks.py b/kuma/api/tasks.py
--- a/kuma/api/tasks.py
+++ b/kuma/api/tasks.py
@@ -137,9 +137,8 @@ def publish(doc_pks, log=None, completion_message=None,
             redirect_url, is_redirect_to_document = redirect
             if is_redirect_to_document:
                 kwargs.update(WebsiteRedirectLocation=redirect_url)
-            else:
-                data = document_api_data(redirect_url=redirect_url)
-                kwargs.update(Body=json.dumps(data))
+            data = document_api_data(redirect_url=redirect_url)
+            kwargs.update(Body=json.dumps(data))
         else:
             data = document_api_data(doc)
             kwargs.update(Body=json.dumps(data))
diff --git a/kuma/api/v1/views.py b/kuma/api/v1/views.py
--- a/kuma/api/v1/views.py
+++ b/kuma/api/v1/views.py
@@ -58,10 +58,13 @@ def doc(request, locale, slug):
 
 
 def get_s3_key(doc=None, locale=None, slug=None,
-               prefix_with_forward_slash=False):
+               prefix_with_forward_slash=False,
+               suffix_file_extension=True):
     if doc:
         locale, slug = doc.locale, doc.slug
     key = reverse('api.v1.doc', args=(locale, slug))
+    if suffix_file_extension:
+        key += '.json'
     if prefix_with_forward_slash:
         # Redirects within an S3 bucket must be prefixed with "/".
         return key
@@ -70,7 +73,11 @@ def get_s3_key(doc=None, locale=None, slug=None,
 
 def get_cdn_key(locale, slug):
     """Given a document's locale and slug, return the "key" for the CDN."""
-    return get_s3_key(locale=locale, slug=slug, prefix_with_forward_slash=True)
+    return get_s3_key(
+        locale=locale,
+        slug=slug,
+        prefix_with_forward_slash=True,
+        suffix_file_extension=False)
 
 
 def get_content_based_redirect(document):
@@ -88,7 +95,10 @@ def get_content_based_redirect(document):
         if redirect_document:
             # This is a redirect to another document.
             return (
-                get_s3_key(redirect_document, prefix_with_forward_slash=True),
+                get_s3_key(
+                    redirect_document,
+                    prefix_with_forward_slash=True,
+                    suffix_file_extension=False),
                 True
             )
         # This is a redirect to non-document page. For now, if it's the home
@@ -156,6 +166,7 @@ def document_api_data(doc=None, redirect_url=None):
             'bodyHTML': doc.get_body_html(),
             'quickLinksHTML': doc.get_quick_links_html(),
             'tocHTML': doc.get_toc_html(),
+            'raw': doc.html,
             'parents': [
                 {
                     'url': d.get_absolute_url(),
diff --git a/kuma/wiki/management/commands/render_document.py b/kuma/wiki/management/commands/render_document.py
--- a/kuma/wiki/management/commands/render_document.py
+++ b/kuma/wiki/management/commands/render_document.py
@@ -36,6 +36,12 @@ def add_arguments(self, parser):
             '--all',
             help='Render ALL documents (rather than by path)',
             action='store_true')
+        parser.add_argument(
+            '--locale',
+            help='Publish ALL documents in this locale (rather than by path)')
+        parser.add_argument(
+            '--not-locale',
+            help='Publish all documents NOT in this locale')
         parser.add_argument(
             '--min-age',
             help='Documents rendered less than this many seconds ago will be'
@@ -54,10 +60,6 @@ def add_arguments(self, parser):
             '--nocache',
             help='Use Cache-Control: no-cache instead of max-age=0',
             action='store_true')
-        parser.add_argument(
-            '--defer',
-            help='Defer rendering by chaining tasks via celery',
-            action='store_true')
         parser.add_argument(
             '--skip-cdn-invalidation',
             help=(
@@ -73,6 +75,7 @@ def handle(self, *args, **options):
         else:
             cache_control = 'max-age=0'
         force = options['force']
+        invalidate_cdn_cache = not options['skip_cdn_invalidation']
 
         if options['all']:
             # Query all documents, excluding those whose `last_rendered_at` is
@@ -83,10 +86,19 @@ def handle(self, *args, **options):
             docs = Document.objects.filter(
                 Q(last_rendered_at__isnull=True) |
                 Q(last_rendered_at__lt=min_render_age))
+            if options['locale']:
+                docs = docs.filter(locale=options['locale'])
+            if options['not_locale']:
+                docs = docs.exclude(locale=options['not_locale'])
             docs = docs.order_by('-modified')
             docs = docs.values_list('id', flat=True)
 
-            self.chain_render_docs(docs, cache_control, base_url, force)
+            self.chain_render_docs(
+                docs,
+                cache_control,
+                base_url,
+                force,
+                invalidate_cdn_cache=invalidate_cdn_cache)
 
         else:
             # Accept page paths from command line, but be liberal
@@ -95,7 +107,7 @@ def handle(self, *args, **options):
             paths = options['paths']
             if not paths:
                 raise CommandError('Need at least one document path to render')
-            invalidate_cdn_cache = not options['skip_cdn_invalidation']
+
             for path in paths:
                 if path.startswith('/'):
                     path = path[1:]


Test patch:
diff --git a/kuma/api/tests/test_tasks.py b/kuma/api/tests/test_tasks.py
--- a/kuma/api/tests/test_tasks.py
+++ b/kuma/api/tests/test_tasks.py
@@ -6,7 +6,8 @@
 import pytest
 
 from kuma.api.tasks import publish, request_cdn_cache_invalidation, unpublish
-from kuma.api.v1.views import document_api_data, get_s3_key
+from kuma.api.v1.views import (
+    document_api_data, get_content_based_redirect, get_s3_key)
 from kuma.wiki.templatetags.jinja_helpers import absolutify
 
 
@@ -102,13 +103,15 @@ def test_publish_redirect(get_s3_bucket_mock, root_doc, redirect_doc):
     log_mock = mock.Mock()
     get_s3_bucket_mock.return_value = s3_bucket_mock = get_mocked_s3_bucket()
     publish([redirect_doc.pk], log=log_mock)
+
     s3_bucket_mock.put_object.assert_called_once_with(
         ACL='public-read',
         Key=get_s3_key(redirect_doc),
         WebsiteRedirectLocation=get_s3_key(
-            root_doc, prefix_with_forward_slash=True),
+            root_doc, prefix_with_forward_slash=True, suffix_file_extension=False),
         ContentType='application/json',
-        ContentLanguage=redirect_doc.locale
+        ContentLanguage=redirect_doc.locale,
+        Body=json.dumps(document_api_data(None, redirect_url=get_content_based_redirect(redirect_doc)[0]))
     )
     log_mock.info.assert_called_once_with('Published S3 Object #1')
 
@@ -178,9 +181,10 @@ def test_publish_multiple(get_s3_bucket_mock, root_doc, redirect_doc,
             ACL='public-read',
             Key=get_s3_key(redirect_doc),
             WebsiteRedirectLocation=get_s3_key(
-                root_doc, prefix_with_forward_slash=True),
+                root_doc, prefix_with_forward_slash=True, suffix_file_extension=False),
             ContentType='application/json',
-            ContentLanguage=redirect_doc.locale
+            ContentLanguage=redirect_doc.locale,
+            Body=json.dumps(document_api_data(redirect_url=get_content_based_redirect(redirect_doc)[0])),
         ),
         mock.call(
             ACL='public-read',
diff --git a/kuma/api/v1/tests/test_views.py b/kuma/api/v1/tests/test_views.py
--- a/kuma/api/v1/tests/test_views.py
+++ b/kuma/api/v1/tests/test_views.py
@@ -13,7 +13,7 @@
 
 def test_get_s3_key(root_doc):
     locale, slug = root_doc.locale, root_doc.slug
-    expected_key = 'api/v1/doc/{}/{}'.format(locale, slug)
+    expected_key = 'api/v1/doc/{}/{}.json'.format(locale, slug)
     assert (
         get_s3_key(root_doc) == get_s3_key(locale=locale, slug=slug) ==
         expected_key
@@ -37,7 +37,12 @@ def test_get_content_based_redirect(root_doc, redirect_doc, redirect_to_self,
         expected = None
     elif case == 'redirect':
         doc = redirect_doc
-        expected = (get_s3_key(root_doc, prefix_with_forward_slash=True), True)
+        expected = (
+            get_s3_key(
+                root_doc,
+                prefix_with_forward_slash=True,
+                suffix_file_extension=False),
+            True)
     elif case == 'redirect-to-self':
         doc = redirect_to_self
         expected = None