Repository: borgbackup/borg

Issue:
online docs: search results incomplete
the docs search (sphinx / readthedocs.io) is a bit broken. searching for "append-only" brings up everything except the section "Append-only mode", which would be the most important result.

http://borgbackup.readthedocs.io/en/stable/search.html?q=append-only&check_keywords=yes&area=default#

http://borgbackup.readthedocs.io/en/stable/usage.html#append-only-mode

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/36995508-online-docs-search-results-incomplete)

Note: this bug is likely not in borg repo, but located in sphinx project or on readthedocs.

https://github.com/sphinx-doc/sphinx/issues/2989 related sphinx issue



Discussion:
Workaround: download the PDF (or epub?) there and search in there.

Note: it also does not find the heading if one searches for "append only": 

http://borgbackup.readthedocs.io/en/stable/search.html?q=append+only&check_keywords=yes&area=default#

Increased the bounty. It is quite important that docs have a working search function.
Hi, I found the issue, on rebuilds the searchindex.js gets re generated with only the updated pages.
The temp solution is always force clean builds. I reported the bug here: https://github.com/sphinx-doc/sphinx/issues/3270
@Blendify great find!

Sadly, it does not explain / solve the issue described above, which seems to be special because the search term is in a heading and might be also influenced by tokenization of the docs text and/or the search term.
This is from 1.0.9 docs build (which was triggered automatically via hook, no manual intervention):
```
Running Sphinx v1.3.5
making output directory...
loading translations [en]... done
loading pickled environment... not yet created
building [mo]: targets for 0 po files that are out of date
building [readthedocs]: targets for 15 source files that are out of date
updating environment: 15 added, 0 changed, 0 removed
reading sources... [  6%] api
reading sources... [ 13%] authors
reading sources... [ 20%] book
reading sources... [ 26%] changes
reading sources... [ 33%] deployment
reading sources... [ 40%] development
reading sources... [ 46%] faq
reading sources... [ 53%] index
reading sources... [ 60%] installation
reading sources... [ 66%] internals
reading sources... [ 73%] introduction
reading sources... [ 80%] quickstart
reading sources... [ 86%] resources
reading sources... [ 93%] support
reading sources... [100%] usage
```
Notice: all "out of date" and "added", 0 "changed".

So it seems (as expected) that on RTFD they always start "clean".
Hmm, guess I found what the problems are:

- it seems to list only 1 (the first) hit in a file (e.g. in usage.html) on the search results page.
- it doesn't rank heading hits higher than paragraph hits (maybe it does no ranking at all?)
- above issues combined mean that if the heading hit is not the first hit within a file, it is not shown.

Another usability issue is that for results it shows, the link only links to the file/url (e.g. to usage.html), it does not link to the section / heading where it found the search term. It activates highlighting there, but for long pages like our usage.html this is less helpful than a more precise link would be.


Have you tried using sphinx 1.5.1? A lot of changes have been made to improve the search since then. Since you are using the RTD theme you will have to add a pip requirements file with: abfa98539a2bfc44198a9ca8c2f16efe84cc4d26

```
git+https://github.com/snide/sphinx_rtd_theme@abfa98539a2bfc44198a9ca8c2f16efe84cc4d26
```

Optionally what I did was add all the dependences to the file so that versions are the same across computers/servers. That would look like this:

```
alabaster==0.7.9
Babel==2.3.4
click==6.6
docutils==0.13.1
imagesize==0.7.1
Jinja2==2.8
MarkupSafe==0.23
polib==1.0.7
Pygments==2.1.3
pytz==2016.10
requests==2.12.3
six==1.10.0
snowballstemmer==1.2.1
Sphinx==1.5.1
sphinx-intl==0.9.9
git+https://github.com/snide/sphinx_rtd_theme@abfa98539a2bfc44198a9ca8c2f16efe84cc4d26
```
_Note that the above list also includes some modules meant for doing translations with sphinx._
@Blendify so, are you saying that by using that requirements file, readthedocs.org will use a more recent sphinx and everything will still work when they build / publish our docs?

I always wondered why they use an old version, but suspected that a new version maybe breaks something there.
Yes there are options for this found here: https://readthedocs.org/dashboard/borgbackup/advanced/
@Blendify thanks, I'll try that. I knew that option but always thought it is for other project requirements and didn't think I could just install a newer sphinx via it... :-D
Note that the theme needs access to git which I do not know how that works with rtd.
http://borgbackup.readthedocs.io/en/latest/ currently points to our docs built with sphinx 1.5.1. I didn't see much change in search quality.

I could not make it use the theme from git - at least not the css, it always uses its standard css.
Well good, I feel much less stupid now :+1: 

My recent experience is described:

When searching for certain terms, I expected results for each occurrence of the searched-for term.  In my case, I tried searching "borg help patterns" and received [two results](http://borgbackup.readthedocs.io/en/latest/search.html?q=borg+help+patterns&check_keywords=yes&area=default).  The results point to the Quick Start and Usage pages. When clicking the result for "Usage", you are taken to the top of the Usage documentation, with all search terms highlighted throughout the document.  In my case, all occurrences of the words "borg", "help", and "patterns" were highlighted without any sort of weighting given to the results. Not seeing the expected results immediately, I *wrongly* assumed what I was searching for wasn't in the docs. Perhaps if I had spent more time, I would have found what I was looking for. Once TW pointed out that what I wanted was indeed in the docs, I spent more time trying to find the information until I found it. Going forward, I can be cognizant of the search limitations.

What I expected, initially, was a search function similar to that provided by Github, and pretty much every other search engine.  All occurrences would be returned in the search results, ordered with the most relevant results at the top. I also expected the search results would take me directly to the section/term in the target documentation.

I'm not sure this is a problem that can be fixed from Borg. This seems like a usability issue with RTD.
More evidence that this needs to be resolved by RTD:
http://docs.readthedocs.io/en/latest/search.html?q=python.setup_py_install&check_keywords=yes&area=default

What I expected to receive was a result pointing to: http://docs.readthedocs.io/en/latest/yaml-config.html#python-setup-py-install

Instead, I get a result pointing to http://docs.readthedocs.io/en/latest/yaml-config.html
To be useful, I would need to scan through the yaml-config page until I *hopefully* found what I was looking for.

In the mean time, Borg can update its docs to keep each page relatively short, thereby increasing the chances that someone will find the result they are searching for.  In the case of my Borg search described previously, I am directed to the Usage page, which contains a great deal of information on a single page.
Yes, esp. splitting the huge usage "page" with all commands into 1 page per command might be useful.
Possible related issues

https://github.com/rtfd/readthedocs.org/issues/2328
https://github.com/rtfd/readthedocs.org/issues/2457

While reviewing RTFD issues, most of the issues which are similar seem to be related to mkdocs... which, iirc, borg is using Sphinx.
> Have you tried using sphinx 1.5.1?

I always build build the docs locally with a fairly recent version. When I looked into this again a few days ago, I used Sphinx 1.5.3. I did not notice improvements in the search.

--

I noticed that when I search the Python docs I get useful results:

![2017-05-22-165218_3840x1440_scrot](https://cloud.githubusercontent.com/assets/1330164/26314789/54f1cc88-3f0f-11e7-8fb6-cf5b1664fe29.png)

![2017-05-22-165312_3840x1440_scrot](https://cloud.githubusercontent.com/assets/1330164/26314792/56aa4398-3f0f-11e7-9129-0d31d801049a.png)

Notice all these results with special markup around them. These are not generic text matches.

Compare:

![2017-05-22-165406_3840x1440_scrot](https://cloud.githubusercontent.com/assets/1330164/26314818/68883a0c-3f0f-11e7-92e1-28a04df0b03b.png)

I was unable to reproduce the Python docs result using index directives and roles. Achieving that would probably resolve this issue, in other words, having the search consider `borg` commands and such like it considers Python classes, functions and other API things.
Plus I don't really understand why it doesn't work, since e.g.

```
.. function:: pyfunc()

   Describes a Python function.
```

works perfectly and shows up in the search results like you'd expect, while index entries defined via `.. index::` or `:index:...` simply do not show up in search results, ever. But they are added to the index. It's very confustrating.

The situation is the same with [these](http://www.sphinx-doc.org/en/stable/domains.html#the-standard-domain). They don't work with the search. Is the Python and C stuff special-cased for the search?

Resolving patch:
diff --git a/docs/borg_domain.py b/docs/borg_domain.py
new file mode 100644
--- /dev/null
+++ b/docs/borg_domain.py
@@ -0,0 +1,102 @@
+from sphinx import addnodes
+from sphinx.domains import Domain, ObjType
+from sphinx.locale import l_, _
+from sphinx.directives import ObjectDescription
+from sphinx.domains.std import ws_re
+
+
+class BorgObject(ObjectDescription):
+    indextemplate = l_('%s')
+    parse_node = None  # type: Callable[[GenericObject, BuildEnvironment, unicode, addnodes.desc_signature], unicode]  # NOQA
+
+    def handle_signature(self, sig, signode):
+        # type: (unicode, addnodes.desc_signature) -> unicode
+        pass
+
+    def add_target_and_index(self, name, sig, signode):
+        # type: (str, str, addnodes.desc_signature) -> None
+        #                  ^ ignore this one, don't insert any markup.
+        # v- the human text               v- the target name
+        # "borg key change-passphrase" -> "borg-key-change-passphrase"
+        del name  # ignored
+        targetname = sig.replace(' ', '-')
+        if self.indextemplate:
+            colon = self.indextemplate.find(':')
+            if colon != -1:
+                indextype = self.indextemplate[:colon].strip()
+                indexentry = self.indextemplate[colon + 1:].strip() % (sig,)
+            else:
+                indextype = 'single'
+                indexentry = self.indextemplate % (sig,)
+            self.indexnode['entries'].append((indextype, indexentry, targetname, '', None))
+        self.env.domaindata['borg']['objects'][targetname] = self.env.docname, self.objtype, sig
+
+    def run(self):
+        super().run()
+        return [self.indexnode]
+
+
+class BorgCommand(BorgObject):
+    """
+    Inserts an index entry and an anchor for a borg command.
+
+    For example, the following snippet creates an index entry about the "borg foo-and-bar"
+    command as well as a "borg-foo-and-bar" anchor (id).
+
+        .. borg:command:: borg foo-and-bar
+    """
+
+    indextemplate = l_('%s (command)')
+
+
+class BorgEnvVar(BorgObject):
+    """
+    Inserts an index entry and an anchor for an environment variable.
+    (Currently not used)
+    """
+
+    indextemplate = l_('%s (environment variable)')
+
+
+class BorgDomain(Domain):
+    """Land of the Borg."""
+    name = 'borg'
+    label = 'Borg'
+    object_types = {
+        'command':   ObjType(l_('command')),
+        'env_var':   ObjType(l_('env_var')),
+    }
+    directives = {
+        'command': BorgCommand,
+        'env_var': BorgEnvVar,
+    }
+    roles = {}
+    initial_data = {
+        'objects': {},  # fullname -> docname, objtype
+    }
+
+    def clear_doc(self, docname):
+        # required for incremental builds
+        try:
+            del self.data['objects'][docname]
+        except KeyError:
+            pass
+
+    def merge_domaindata(self, docnames, otherdata):
+        # needed due to parallel_read_safe
+        for fullname, (docname, objtype, sig) in otherdata['objects'].items():
+            if docname in docnames:
+                self.data['objects'][fullname] = (docname, objtype, sig)
+
+    def get_objects(self):
+        for refname, (docname, objtype, sig) in list(self.data['objects'].items()):
+            yield sig, sig, objtype, docname, refname, 1
+
+
+def setup(app):
+    app.add_domain(BorgDomain)
+    return {
+        'version': 1,
+        'parallel_read_safe': True,
+        'parallel_write_safe': True,
+    }
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -16,6 +16,7 @@
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 import sys, os
 sys.path.insert(0, os.path.abspath('../src'))
+sys.path.insert(0, os.path.abspath('.'))
 
 from borg import __version__ as sw_version
 
@@ -26,9 +27,7 @@
 # If your documentation needs a minimal Sphinx version, state it here.
 #needs_sphinx = '1.0'
 
-# Add any Sphinx extension module names here, as strings. They can be extensions
-# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = []
+# Extensions are defined at the end of this file.
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -241,7 +240,14 @@
          1),
 ]
 
-extensions = ['sphinx.ext.extlinks', 'sphinx.ext.autodoc', 'sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.viewcode']
+extensions = [
+    'sphinx.ext.extlinks',
+    'sphinx.ext.autodoc',
+    'sphinx.ext.todo',
+    'sphinx.ext.coverage',
+    'sphinx.ext.viewcode',
+    'borg_domain',
+]
 
 extlinks = {
     'issue': ('https://github.com/borgbackup/borg/issues/%s', '#'),
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -258,6 +258,7 @@ def generate_level(self, prefix, parser, Archiver):
                     params = {"command": command,
                               "command_": command.replace(' ', '_'),
                               "underline": '-' * len('borg ' + command)}
+                    doc.write(".. borg:command:: borg {command}\n\n".format(**params))
                     doc.write(".. _borg_{command_}:\n\n".format(**params))
                     doc.write("borg {command}\n{underline}\n::\n\n    borg [common options] {command}".format(**params))
                     self.write_usage(parser, doc)


Test patch: