Repository: readthedocs/readthedocs.org

Issue:
Deprecate `Project.documentation_type`
Our config file is going to support per-version documentation types, so we need to make a few changes to support this logic. See #4486 for some background here. The two points where we are still using `Project.documenation_type` are in the `footer_html` return and in our resolver.

We can't remove the need for the data from this field, as we have some logic that determines file path based on the build backend type. So, we have two options:

### Remove documentation type completely

Is there another way to remove this logic or not depend on a `Version.documentation_type`?

I think we are :-1: on this method as it may not be possible without removing the path logic. If there is a way, this is the easiest though.

### Move to a generated `Version.documentation_type`

A couple options for this field:

* Make it a model field
* Make it a property that uses the last successful `Build.documentation_type` instead

My vote is maybe for option 2. We already use `Build` as a writing point for build metadata. Updating a `documentation_type` from the config would make sense here. We also talked about adding a general `Build.config` for storing the used config on the build. Perhaps this work can be combined here. Perhaps cache this property as well?

Pieces that will be or could be changed here:

- [x] Add a `Build.config` json blob for storing config meta data (separate PR?)
- [x] Make build update `Build.config` json blob
- [x] Make `Version.documentation_type` property
- [ ] Move logic for determining path to `Version`?
- [ ] Tune footer_html template to not use `main_language_project.get_url() + path`, but instead use something closer to `Version.get_url_for(path)`. Logic should get the per-version URL, not kludge together a URL.
- [x] Rely on `self.config` for change in #4655
Broken links in version picker menu for HtmlDir index pages
## Details

* Read the Docs project URL: https://anymail.readthedocs.io/

In Sphinx-HtmlDir docs, the version picker menu has broken links on any page that is generated from a subdir/index.rst. The URLs for such pages look like `.../subdir/`, but the version picker menu incorrectly links to `.../subdir/index/`, which is a 404.

(Fairly certain this used to work correctly.)

## Expected Result

To reproduce:
1. Go to (e.g.) https://anymail.readthedocs.io/en/stable/sending/. (This is a Sphinx-HtmlDir project, the source for this page is [docs/sending/index.rst](https://github.com/anymail/django-anymail/blob/v5.0/docs/sending/index.rst).)
2. Click the green v:stable menu at the bottom of the left sidebar.
3. Try to go to "latest" (or any other version)

Expected Result: The link should be to https://anymail.readthedocs.io/en/latest/sending/ (without the "index" at the end).

## Actual Result

Actual Result: The menu links to https://anymail.readthedocs.io/en/latest/sending/index/ (SORRY: This page does not exist yet.)

<img width="344" alt="version menu with incorrect index link" src="https://user-images.githubusercontent.com/639984/52449172-281a3c80-2aeb-11e9-8918-bbbb648c22f2.png">

Note that on *non*-index pages, the version menu works correctly  (e.g., https://anymail.readthedocs.io/en/stable/sending/django_email/, built from docs/sending/django_email.rst)


Discussion:
Just linking the commit that we'll need to revert later https://github.com/rtfd/readthedocs.org/pull/4486/commits/80e3904f1334c81896fe42909de67fd54046f856
- Add a Build.config json blob for storing config meta data (separate PR?)
- Make build update Build.config json blob

Those looks the same for me p: do you mean on 1) making something like we do with the build output? (store json files instead of the db) and 2) is to store on the db as a json field?

Anyway, I'm +1 for storing this information under a metadata field for the Build model, as we can use that information for other stuff (like show the final configuration used, suggest using a configuration file, etc)
Sorry, no, i meant we need to add a field or fields to Build and the second part is making the API update these pieces on successful build.

If we don't have other metadata, we can also use a one-off field. But I think we're leaning towards storing more metadata (it would help greatly with debugging and UX around build).
So, I re-read and think I get it now p: I'm +1 for using a json field. Do we want to do something like output field (it's saved in other location as a json file)?

If we don't want to complicate things, I'm fine with one field too.
No, we don't want to store this outside the build as we'll be using the data directly. Build command output goes to cold storage eventually as it's really only user facing output and is probably rarely accessed compared to how often we're storing it.

This would just be a json field on build model I think. If there is a reason to query any of this data, that might push us towards another model mechanism, but django-jsonfield at least has some ability to be queried.
So, I was digging a little here about removing `Project.documentation_type` completely.

- Previous projects don't have the config saved (we can do a migration here, just update all previous builds of earch version to the doctumentation_type of the project).
- People that want to build their docs with other than sphinx, have to use a config file.

If we don't want to completely removed it:

- We can have a property in version, wich will default to the project doctype if we don't have the doctype in the config field of the build (or we can still do the migration here).
- Default the doctype to the one from the project and save that to the config (we already do this on master).

Also, for the footer call, I reliaze we need to know the doctype because we want to generate the links to the other docs, as Eric mentioned, querying the config from the build objects could be slow. So, what about passing the doctype to the footer request (I mean saving it in the rtd-data.js file)? Old projects will still need the old logic.

Another idea, why not just send the whole ulr? and replace the lang/version in the api footer? Again, old projects will need to keep the old logic.

For the resolver, honestly I have no idea why do we need to use documentation type there :disappointed:, some locig looks like the one from the footer...

/cc @rtfd/core  
> Also, for the footer call, I reliaze we need to know the doctype because we want to generate the links to the other docs, as Eric mentioned, querying the config from the build objects could be slow

How much is _slow_? We should need some measure here. I suppose that PostgreSQL is optimized to do queries inside a JSON field. In case we consider that it's slower than we want, we can always denormalize the db a little and duplicate this data into a normal field (which will perform fast).
> How much is slow? We should need some measure here. I suppose that PostgreSQL is optimized to do queries inside a JSON field. 

Mostly it's because we are doing a query on a larger table (builds) on each request to the footer API, I'm not worried in other parts of the code (maybe in the resolver?). That's why I suggest saving this data on the `rtd-data.js` file, we alredy save a lot of info for the footer API there.
Caching could be fine here as well, we could set the cache on each build even. This would remove the need to have to update the Version object from a project build.

We could either treat this like a feature flag, or use a date based feature flag to change this UI. Old projects might know how to edit the project documentation type. New projects should be pushed to use the config.
So, I've investigated the resolver, doctype is used inside https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/core/resolver.py#L93-L93

in https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/core/resolver.py#L254-L254

The usage of resolve_path are:

https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/core/views/serve.py

Used to serve docs from /docs (doctype isn't always needed), and we don't used it in production, because we serve from nginx.

https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/redirects/models.py

Used in redirects. Isn't used at all, or it shouldn't, because it only returns a redirect, isn't serving from a file.

https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/projects/tasks.py#L1149-L1149

En tasks (_manage_imported_files). Isn't used at all, because all files are valid files (html and not-html)

https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/projects/tasks.py#L1215-L1221
> Used to serve docs from /docs (doctype isn't always needed), and we don't used it in production, because we serve from nginx.

This is not true for the .com or in local dev. I don't see where we'd need to know the file ending of a file tho, we should just pass it in explicitly. 
Looks like not, https://docs.djangoproject.com/en/1.11/ref/views/#django.views.static.serve it has `show_indexes=False` by default and nginx accepts an uri https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/#x-accel-redirect

I'd say we are safe, I'll do some tests anyway.
Other function where we use this is `readthedocs.core.resolver.resolve`

https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/core/resolver.py#L162-L162

(we call `resolve_path` from here).

And that is used here:

https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/projects/models.py#L24

I checked the code, and we don't need the doctype here

https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/core/views/serve.py#L43-L43

Here there is a place where we are _breaking_ something

https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/core/views/serve.py#L122-L132

Here we are losing redirects like `prject.rtd.io/page/subpage` -> `docs.rtd.io/en/latest/index.html#subpage`

Which in my opinion doesn't look very relevant, and I don't think people are even aware of that...

And the last place

https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/core/templatetags/core_tags.py#L39-L43

This is used in search, so not really sure if this will break something

https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/templates/projects/project_analytics.html#L67-L67

https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/templates/search/elastic_project_search.html#L50-L50

https://github.com/rtfd/readthedocs.org/blob/2bf6e9943c4177d11e125fd077a07f7168fce044/readthedocs/templates/search/elastic_search.html#L144-L144

But at first sight I would say that we always index the name of the file as a full path, so it shouldn't be any problem.
I believe that we ran into the redirect issue re: htmldir builder in the latest CPython devguide build. https://github.com/python/devguide/issues/483
FYI. [Sphinx builder dirhtml in Sphinx 2.0+](https://www.sphinx-doc.org/en/2.0/_modules/sphinx/builders/dirhtml.html) uses the name `dirhtml`. I think [RTD v2 config is using `htmldir`](https://docs.readthedocs.io/en/stable/config-file/v2.html#sphinx-builder). I'm not sure if the two are mapping correctly.
Just an update here, looks like we depend on the doctype on many places. I think we could get rid of it, but it may be a big refactor. So, I think moving the doctype to the version model is a better solution for now.
A source of the problem seems to be [this code](https://github.com/rtfd/readthedocs.org/blob/3.2.1/readthedocs/restapi/views/footer_views.py#L97-L102) in footer_views:

```py
    if page_slug and page_slug != 'index':
        if (
            main_project.documentation_type == 'sphinx_htmldir' or
            main_project.documentation_type == 'mkdocs'
        ):
            path = page_slug + '/'
     # ...
```

page_slug on a sphinx_htmldir subdir index page is "subdir/index" (e.g., "sending/index" in the reported example). I can't find the code that generates page slugs, which is probably the right place to fix this. (That slug should probably be just "sending".)

But an easy workaround here would be `if page_slug.endswith('/index'): page_slug = page_slug[:-6]` just before assigning `path`.


Thanks! If this was working before, I can think of an incompatible change in sphinx, if this wasn't working before, then I think it's a bug. And we want to remove this _magic_ of the documentation type https://github.com/rtfd/readthedocs.org/issues/4638
So, yeah, this is definitely a bug, the page slug for nested dirs is `page/index` while for no-nested pages is just `page`. Not sure if this is a breaking change in sphinx or if this was wrong the whole time, I'll try to check with some previous sphinx versions.
It's very possible I'm remembering wrong that it worked before.

If RTD's `page_slug` is Sphinx's `docname`, it *does* seem like Sphinx's dirhtml builder has had [special](https://github.com/sphinx-doc/sphinx/blob/master/sphinx/builders/dirhtml.py#L37-L38) [handling](https://github.com/sphinx-doc/sphinx/blob/master/sphinx/builders/dirhtml.py#L43-L45) for `docname.endswith(SEP + 'index')` going [back 10 years](https://github.com/sphinx-doc/sphinx/blame/2f61ba876c40bf8c77d74faf372279c3510166bb/sphinx/builders/html.py#L1250-L1267).
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Still valid
@stsewd happy to submit a PR if that would help. I think the fix is in [footer_views.py](https://github.com/rtfd/readthedocs.org/blob/3.4.1/readthedocs/restapi/views/footer_views.py#L96-L97):

```diff
    if page_slug and page_slug != 'index':
        if main_project.documentation_type == 'sphinx_htmldir':
+           if page_slug.endswith('/index'):
+               path = page_slug[:-5]
+           else:
                path = page_slug + '/'
        else:
            path = page_slug + '.html'
    else:
        path = ''
```

If so, I could use some guidance on where to add a test or pointers to similar tests. (test_footer.py? There don't seem to be many tests specific to sphinx_htmldir.)

That should work only if the project is using only `sphinx_htmldir` with all the versions. We are trying to remove the `project.documentation_type` calls https://github.com/rtfd/readthedocs.org/issues/4638

So, not sure if want to keep patching this or fix this for complete with #4638 (it may take some time)
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Resolving patch:
diff --git a/readthedocs/api/v2/views/footer_views.py b/readthedocs/api/v2/views/footer_views.py
--- a/readthedocs/api/v2/views/footer_views.py
+++ b/readthedocs/api/v2/views/footer_views.py
@@ -1,5 +1,7 @@
 """Endpoint to generate footer HTML."""
 
+import re
+
 from django.conf import settings
 from django.shortcuts import get_object_or_404
 from django.template import loader as template_loader
@@ -142,13 +144,12 @@ def _get_context(self):
         version = self._get_version()
 
         page_slug = self.request.GET.get('page', '')
+        path = ''
         if page_slug and page_slug != 'index':
-            if main_project.documentation_type == 'sphinx_htmldir':
-                path = page_slug + '/'
+            if version.documentation_type == 'sphinx_htmldir':
+                path = re.sub('/index$', '', page_slug) + '/'
             else:
                 path = page_slug + '.html'
-        else:
-            path = ''
 
         context = {
             'project': project,
diff --git a/readthedocs/config/config.py b/readthedocs/config/config.py
--- a/readthedocs/config/config.py
+++ b/readthedocs/config/config.py
@@ -685,8 +685,6 @@ def validate(self):
         self.validate_doc_types()
         self._config['mkdocs'] = self.validate_mkdocs()
         self._config['sphinx'] = self.validate_sphinx()
-        # TODO: remove later
-        self.validate_final_doc_type()
         self._config['submodules'] = self.validate_submodules()
         self.validate_keys()
 
@@ -965,29 +963,6 @@ def validate_sphinx(self):
 
         return sphinx
 
-    def validate_final_doc_type(self):
-        """
-        Validates that the doctype is the same as the admin panel.
-
-        This a temporal validation, as the configuration file should support per
-        version doctype, but we need to adapt the rtd code for that.
-        """
-        dashboard_doctype = self.defaults.get('doctype', 'sphinx')
-        if self.doctype != dashboard_doctype:
-            error_msg = (
-                'Your project is configured as "{}" in your admin dashboard,'
-            ).format(self.builders_display[dashboard_doctype])
-
-            if dashboard_doctype == 'mkdocs' or not self.sphinx:
-                error_msg += ' but there is no "{}" key specified.'.format(
-                    'mkdocs' if dashboard_doctype == 'mkdocs' else 'sphinx',
-                )
-            else:
-                error_msg += ' but your "sphinx.builder" key does not match.'
-
-            key = 'mkdocs' if self.doctype == 'mkdocs' else 'sphinx.builder'
-            self.error(key, error_msg, code=INVALID_KEYS_COMBINATION)
-
     def validate_submodules(self):
         """
         Validates the submodules key.


Test patch:
diff --git a/readthedocs/config/tests/test_config.py b/readthedocs/config/tests/test_config.py
--- a/readthedocs/config/tests/test_config.py
+++ b/readthedocs/config/tests/test_config.py
@@ -1511,9 +1511,6 @@ def test_sphinx_builder_default(self):
         build.validate()
         build.sphinx.builder == 'sphinx'
 
-    @pytest.mark.skip(
-        'This test is not compatible with the new validation around doctype.',
-    )
     def test_sphinx_builder_ignores_default(self):
         build = self.get_build_config(
             {},
@@ -1686,28 +1683,6 @@ def test_mkdocs_fail_on_warning_check_default(self):
         build.validate()
         assert build.mkdocs.fail_on_warning is False
 
-    def test_validates_different_filetype_mkdocs(self):
-        build = self.get_build_config(
-            {'mkdocs': {}},
-            {'defaults': {'doctype': 'sphinx'}},
-        )
-        with raises(InvalidConfig) as excinfo:
-            build.validate()
-        assert excinfo.value.key == 'mkdocs'
-        assert 'configured as "Sphinx Html"' in str(excinfo.value)
-        assert 'there is no "sphinx" key' in str(excinfo.value)
-
-    def test_validates_different_filetype_sphinx(self):
-        build = self.get_build_config(
-            {'sphinx': {}},
-            {'defaults': {'doctype': 'sphinx_htmldir'}},
-        )
-        with raises(InvalidConfig) as excinfo:
-            build.validate()
-        assert excinfo.value.key == 'sphinx.builder'
-        assert 'configured as "Sphinx HtmlDir"' in str(excinfo.value)
-        assert 'your "sphinx.builder" key does not match' in str(excinfo.value)
-
     def test_submodule_defaults(self):
         build = self.get_build_config({})
         build.validate()
diff --git a/readthedocs/rtd_tests/tests/test_config_integration.py b/readthedocs/rtd_tests/tests/test_config_integration.py
--- a/readthedocs/rtd_tests/tests/test_config_integration.py
+++ b/readthedocs/rtd_tests/tests/test_config_integration.py
@@ -781,9 +781,6 @@ def test_sphinx_builder(
 
         get_builder_class.assert_called_with(result)
 
-    @pytest.mark.skip(
-        'This test is not compatible with the new validation around doctype.',
-    )
     @patch('readthedocs.projects.tasks.get_builder_class')
     def test_sphinx_builder_default(
             self, get_builder_class, checkout_path, tmpdir,
diff --git a/readthedocs/rtd_tests/tests/test_footer.py b/readthedocs/rtd_tests/tests/test_footer.py
--- a/readthedocs/rtd_tests/tests/test_footer.py
+++ b/readthedocs/rtd_tests/tests/test_footer.py
@@ -143,6 +143,60 @@ def test_not_show_edit_on_github(self):
         self.assertIn('View', response.data['html'])
         self.assertNotIn('Edit', response.data['html'])
 
+    def test_index_pages_sphinx_htmldir(self):
+        version = self.pip.versions.get(slug=LATEST)
+        version.documentation_type = 'sphinx_htmldir'
+        version.save()
+
+        # A page with slug 'index' should render like /en/latest/
+        self.url = (
+            reverse('footer_html') +
+            f'?project={self.pip.slug}&version={self.latest.slug}&page=index&docroot=/'
+        )
+        response = self.render()
+        self.assertIn('/en/latest/', response.data['html'])
+        self.assertNotIn('/en/latest/index.html', response.data['html'])
+
+        # A page with slug 'foo/index' should render like /en/latest/foo/
+        self.url = (
+            reverse('footer_html') +
+            f'?project={self.pip.slug}&version={self.latest.slug}&page=foo/index&docroot=/'
+        )
+        response = self.render()
+        self.assertIn('/en/latest/foo/', response.data['html'])
+        self.assertNotIn('/en/latest/foo.html', response.data['html'])
+        self.assertNotIn('/en/latest/foo/index.html', response.data['html'])
+
+        # A page with slug 'foo/bar' should render like /en/latest/foo/bar/
+        self.url = (
+            reverse('footer_html') +
+            f'?project={self.pip.slug}&version={self.latest.slug}&page=foo/bar&docroot=/'
+        )
+        response = self.render()
+        self.assertIn('/en/latest/foo/bar/', response.data['html'])
+        self.assertNotIn('/en/latest/foo/bar.html', response.data['html'])
+        self.assertNotIn('/en/latest/foo/bar/index.html', response.data['html'])
+
+        # A page with slug 'foo/bar/index' should render like /en/latest/foo/bar/
+        self.url = (
+            reverse('footer_html') +
+            f'?project={self.pip.slug}&version={self.latest.slug}&page=foo/bar/index&docroot=/'
+        )
+        response = self.render()
+        self.assertIn('/en/latest/foo/bar/', response.data['html'])
+        self.assertNotIn('/en/latest/foo/bar.html', response.data['html'])
+        self.assertNotIn('/en/latest/foo/bar/index.html', response.data['html'])
+
+        # A page with slug 'foo/index/bar' should render like /en/latest/foo/index/bar/
+        self.url = (
+            reverse('footer_html') +
+            f'?project={self.pip.slug}&version={self.latest.slug}&page=foo/index/bar&docroot=/'
+        )
+        response = self.render()
+        self.assertIn('/en/latest/foo/index/bar/', response.data['html'])
+        self.assertNotIn('/en/latest/foo/index/bar.html', response.data['html'])
+        self.assertNotIn('/en/latest/foo/index/bar/index.html', response.data['html'])
+
 
 class TestFooterHTML(BaseTestFooterHTML, TestCase):