Repository: readthedocs/readthedocs.org

Issue:
ReadTheDocs search and MkDocs
(Sorry for adding this as an issue @ericholscher, but we keep missing each other on IRC. If there is a better place, let me know.)

We added the `mkdocs json` command to MkDocs for RTD's to use in serverside search. Since then I have been [working on clientside search](https://github.com/tomchristie/mkdocs/pull/222) support. This means we generate a JSON index in a slightly different format to work with Tipue search and thus we will potentially have two different JSON output formats which is a bit of a pain.

The Tipue format looks like this: https://github.com/Tipue/Tipue-Search/blob/master/tipuesearch/tipuesearch_content.js . The key difference is that it is one file containing all pages and the key names are different but otherwise I think it contains pretty much the same data.

So, my question is, can RTD be adapted to use this format so we can drop the specific json command from MkDocs? If so, I'd be happy to make the change in the ReadTheDocs code.



Discussion:
Seems reasonable. Our code for that is here.

https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/search/utils.py#L15

https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/tasks.py#L408-L417

https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/tasks.py#L656-L657

Not super great logic/code, mostly left over from how we do Sphinx.
Definitely happy to have someone else write it :)

On Sat, Dec 13, 2014 at 8:05 AM, Dougal Matthews notifications@github.com
wrote:

> (Sorry for adding this as an issue @ericholscher
> https://github.com/ericholscher, but we keep missing each other on IRC.
> If there is a better place, let me know.)
> 
> We added the mkdocs json command to MkDocs for RTD's to use in serverside
> search. Since then I have been working on clientside search
> https://github.com/tomchristie/mkdocs/pull/222 support. This means we
> generate a JSON index in a slightly different format to work with Tipue
> search and thus we will potentially have two different JSON output formats
> which is a bit of a pain.
> 
> The Tipue format looks like this:
> https://github.com/Tipue/Tipue-Search/blob/master/tipuesearch/tipuesearch_content.js
> . The key difference is that it is one file containing all pages and the
> key names are different but otherwise I think it contains pretty much the
> same data.
> 
> So, my question is, can RTD be adapted to use this format so we can drop
> the specific json command from MkDocs? If so, I'd be happy to make the
> change in the ReadTheDocs code.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/rtfd/readthedocs.org/issues/1088.

## 

Eric Holscher
Maker of the internet residing in Portland, Or
http://ericholscher.com

Is this still relevant? `mkdocs json` has since become deprecated...

Yes, I believe this is relevant. We have deprecated it in MkDocs but we won't remove it while RTD still depends on it.

Fair enough. Sorry, I was mislead but I'm grasping at straws trying to figure out why the search for _some_ MkDocs projects works on RTD but not for others (#2020).

No problem. I'm not sure how to help you with that issue, I'm not familiar with RTD's search.

It appears that the form action for the searchbox is being injected with this javascript: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/core/static-src/core/js/doc-embed/mkdocs.js.

It looks like it's just building the wrong query, along the lines of: `https://readthedocs.org/search/?q=bar&check_keywords=yes&area=default&project=foo&version=X.y&type=file`. Either of these paths actually work though:
- https://readthedocs.org/projects/foo/search/?q=bar
- http://foo.readthedocs.io/en/X.y/search.html?q=bar

Thanks for the hint. The first doesn't work for me but the second does:
- https://readthedocs.org/projects/nodemcu/search/?q=docker -> NOK, no results
- http://nodemcu.readthedocs.io/en/dev/search.html?q=docker -> OK

The easiest work-around for this is overriding the searchbox template like so: https://github.com/aegir-project/documentation/compare/2884f80...3a44eec1

Note the change in the form's `id` (e.g. `rtd-search-form-alt`). This stops the RTD JS from breaking the default search functionality.

The fact that your work-around has the desired affect means that RTD by default replaces the MkDocs RTD theme? The original theme's `searchbox.html` isn't so different from yours: https://github.com/mkdocs/mkdocs/blob/master/mkdocs/themes/readthedocs/searchbox.html. So, the custom theme kinda re-introduces the original theme?

RTD isn't replacing the mkdocs theme, just altering it with some javascript. As per http://www.mkdocs.org/user-guide/custom-themes/#creating-a-custom-theme, I just copied the default mkdocs RTD theme's searchbox.html, and altered the form id, so that the JS would no longer have any effect.

@ergonlogic the only "problem" with your nifty custom theme approach I found so far is that RTD changes the implementation of the footer fly-out menu it adds. While it normally sits in the lower left corner it is moved to the lower right and becomes detached once you declare a custom theme in `mkdocs.yml`.
An alternative is to add the following JavaScript function to your `extra.js`. It reverts the changes RTD applies to the search form:

``` javascript
  function fixSearch() {
    var target = document.getElementById('rtd-search-form');
    var config = {attributes: true, childList: true};

    var observer = new MutationObserver(function(mutations) {
      // if it isn't disconnected it'll loop infinitely because the observed element is modified
      observer.disconnect();
      var form = $('#rtd-search-form');
      form.empty();
      form.attr('action', 'https://' + window.location.hostname + '/en/' + determineSelectedBranch() + '/search.html');
      $('<input>').attr({
        type: "text",
        name: "q",
        placeholder: "Search docs"
      }).appendTo(form);
    });
    // don't run this outside RTD hosting
    if (window.location.origin.indexOf('readthedocs') > -1) {
      observer.observe(target, config);
    }
  }
```

Thanks you @marcelstoer the work-around you implemented in https://github.com/nodemcu/nodemcu-firmware/commit/7dd89dd15ef993c9740ba4d4361e6be8edd1284b is working for me!

The latest mkdocs (0.17) removes the json output feature RTD relies on to index, so builds are broken completely with the latest version. It would be great to get someone that knows more about mkdocs maintaining these pieces, we don't have the bandwidth.

Raised in #3174
That is odd because earlier in this issue @d0ugal stated

> We have deprecated it in MkDocs but we won't remove it while RTD still depends on it.
I don’t have the bandwidth for MkDocs now. So I guess that goal stopped with me. I’d recommend pinning at the previous release for now.
Hi,

This issue seems to have been fixed (and/or worked around) by #3361 and #3496, I can confirm the ReadTheDocs search module works properly with MkDocs for the [Shaarli Documentation](https://shaarli.readthedocs.io/en/master/) (ref. https://github.com/shaarli/Shaarli/issues/1033)

Thanks @d0ugal, @ericholscher and @humitos :)
@virtualtam I'm not very into mkdocs nor search, but, does your last comment mean that we need to close this issue or there is something else missing to track here?

(thanks for the update, though!)
@humitos from a user's perspective, the search feature works for documentation generated with MkDocs and hosted on RTD, so I would suggest closing this issue (unless further changes are needed).
It sounds like this is resolved.
I believe this Issue should be open as we are not indexing MkDocs in Elasticsearch index.
But it uses the default search from mkdocs (I think this issue was about the search in the docs, not in the rtd.org site?)
@stsewd Yeah. currently it uses built in search of mkdocs. But it would be better if we could index the content to our elasticsearch index so we can provide better search result to our users.
@safwanrahman maybe this interest you https://github.com/rtfd/readthedocs.org/issues/4205
> But it would be better if we could index the content to our elasticsearch index so we can provide better search result to our users

IIRC, the problem we have with MkDocs is there is no way to generate a context to create the `.json` file as we do with Sphinx. The limitation was on MkDocs side, not on us.

I may be wrong, though.
@humitos The json file is created with the HTML file by default. But the structure is different. We need to port our code to index the json file into our Elasticsearch.
We need a way to dump mkdocs HTML unthemed, which is the main problem. It needs to be consistent across themes and builds. 
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This is still valid, but blocked and/or needs a design decision
This issue is related to https://github.com/rtfd/readthedocs.org/issues/4588
This isn't fully closed. We are going to be indexing search data for mkdocs since next week, but not yet overriding the mkdocs search for ours.

Resolving patch:
diff --git a/readthedocs/doc_builder/backends/mkdocs.py b/readthedocs/doc_builder/backends/mkdocs.py
--- a/readthedocs/doc_builder/backends/mkdocs.py
+++ b/readthedocs/doc_builder/backends/mkdocs.py
@@ -253,6 +253,12 @@ def generate_rtd_data(self, docs_dir, mkdocs_config):
             'commit': self.version.project.vcs_repo(self.version.slug).commit,
             'global_analytics_code': settings.GLOBAL_ANALYTICS_CODE,
             'user_analytics_code': analytics_code,
+            'features': {
+                'docsearch_disabled': (
+                    not self.project.has_feature(Feature.ENABLE_MKDOCS_SERVER_SIDE_SEARCH)
+                    or self.project.has_feature(Feature.DISABLE_SERVER_SIDE_SEARCH)
+                )
+            },
         }
 
         data_ctx = {
diff --git a/readthedocs/doc_builder/backends/sphinx.py b/readthedocs/doc_builder/backends/sphinx.py
--- a/readthedocs/doc_builder/backends/sphinx.py
+++ b/readthedocs/doc_builder/backends/sphinx.py
@@ -181,6 +181,7 @@ def get_config_params(self):
             'dont_overwrite_sphinx_context': self.project.has_feature(
                 Feature.DONT_OVERWRITE_SPHINX_CONTEXT,
             ),
+            'docsearch_disabled': self.project.has_feature(Feature.DISABLE_SERVER_SIDE_SEARCH),
         }
 
         finalize_sphinx_context_data.send(
diff --git a/readthedocs/projects/models.py b/readthedocs/projects/models.py
--- a/readthedocs/projects/models.py
+++ b/readthedocs/projects/models.py
@@ -1589,6 +1589,8 @@ def add_features(sender, **kwargs):
     SKIP_SYNC_BRANCHES = 'skip_sync_branches'
     CACHED_ENVIRONMENT = 'cached_environment'
     LIMIT_CONCURRENT_BUILDS = 'limit_concurrent_builds'
+    DISABLE_SERVER_SIDE_SEARCH = 'disable_server_side_search'
+    ENABLE_MKDOCS_SERVER_SIDE_SEARCH = 'enable_mkdocs_server_side_search'
     FORCE_SPHINX_FROM_VENV = 'force_sphinx_from_venv'
     LIST_PACKAGES_INSTALLED_ENV = 'list_packages_installed_env'
     VCS_REMOTE_LISTING = 'vcs_remote_listing'
@@ -1669,6 +1671,14 @@ def add_features(sender, **kwargs):
             LIMIT_CONCURRENT_BUILDS,
             _('Limit the amount of concurrent builds'),
         ),
+        (
+            DISABLE_SERVER_SIDE_SEARCH,
+            _('Disable server side search'),
+        ),
+        (
+            ENABLE_MKDOCS_SERVER_SIDE_SEARCH,
+            _('Enable server side search for MkDocs projects'),
+        ),
         (
             FORCE_SPHINX_FROM_VENV,
             _('Force to use Sphinx from the current virtual environment'),


Test patch: