Repository: borgbackup/borg

Issue:
make a better / real man page
README.Debian from the debian package tells about quite some difficulties about creating a man page.

I think this should be solved upstream (== by us), also to make it easier for other linux distributions.

Debian packaging is discussed here:

https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1355849.html

The solution for creating the manpage should be automated.

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/26801934-make-a-real-man-page)



Discussion:
more accurate URL for the packaging work is: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=792096

i was able to generate man pages by formatting argparse usage using some shims in monkeysign, initially inspired by:

https://andialbrecht.wordpress.com/2009/03/17/creating-a-man-page-with-distutils-and-optparse/

original source: https://crunchyfrog.googlecode.com/hg/utils/command/build_manpage.py

current implementation: https://sources.debian.net/src/monkeysign/2.0.2/monkeysign/documentation.py/#L13

this would need to be ported to argparse (easy) and eventually to click (harder, see https://github.com/mitsuhiko/click/issues/434 and #251)

i gave that my best shot, and argparse is just a nightmare to handle, particularly with our sub-parsers and everything. i would suggest we wait for the click port in #251 before spending any more time on this.

my current work is in my [man-builder branch](https://github.com/anarcat/borg/commits/man-builder) if anyone is masochistic enough to continue this nonsense.

what about using rst2man?

In syncthing we do it like this:
- http://docs.syncthing.net/users/syncthing.html
- https://github.com/syncthing/docs/blob/master/conf.py#L251-L291

@LocutusOfBorg would be certainly easier than the curent approach. however we'd need to split the manpages better: right now the `usage.rst.inc` files is all in one file, so we can't use rst2man to generate multiple manpages.

but i think it makes perfect sense, especially considering how messy #384 has become. we could revert all the mock and `have_cython` stuff and simply generate the `rst` files statically. then we'd have a much better control over the content of those files and could generate manpages, which would be readable directly online here (in github) as well.

@ThomasWaldmann would that satisfy the bounty you posted?

i did the above suggestion in #408. the next step here will be to split the usage file into multiple parts so that rst2man (or whatever sphinx uses for the `man_pages` variable) to kick in.

@rumpelsepp thanks for the suggestion, looks like the way to go now...

@rumpelsepp how do you keep those rst files in sync with the code? by hand?

i built #409 to fix this.

remaining todos after the PR:
- hook manpages into `setup.py`
- turn `usage.rst` into a `borg.1` manpage so the docs there can also be read online

@anarcat: Yeah, by hand. Since you "only" have to document the command line interface (which might not be changed that often), it is not that big deal to keep that in sync. One should keep in mind to always update documentation in a PR as well when the CLI changes.

And also, I suggest to structure the manpages like git does: `borg-create(1)`, `borg-prune(1)`, ... All possible subcommands in one manpage is a bit of a pain to read.

@rumpelsepp then you may find it interesting to look at the `build_usage` in `setup.py` in #409 - it generates the `rst` sources for each manpage automatically. it lacks per-option markup, but it's otherwise quite useful.

and i agree about `borg-create(1)`, just need to convince @ThomasWaldmann now. :p

I guess we can have borg-*, but there also should be a borg manpage with the basics and to refer to the others.

yeah, that's what i was hoping to do with the `usage.rst` file, but since it has that table of contents on top, it actually includes all the other manpages in the rendering... so some more work required.

also note that, in debian, doing "man borg create" will find the "borg-create(1)" manpage.

Hi, 
If you are using argparse, you might consider using this:
https://github.com/oz123/man-utils

I am the author of this, and I would like to get some feed back.  I basically adds a method to your setup.py which you can use to create the man page. 
All it does is parse your argparse...

well, i guess the first feedback i will give you is that i have no idea how to use this in our project. the only documentation i could find says to `import man-utils`, but where should i do that? near main? in setup.py? you mention a method in setup.py, but that's not documented either...

it looks like the goods are in https://github.com/oz123/man-utils/blob/master/manutils/manutils.py

@ThomasWaldmann that code looks pretty useful, but it would require sucking it all into setup.py (as is done in the author's original project, [pwman3](https://github.com/pwman3/pwman3/blob/develop/setup.py)) - or moving it to an import, say in borg/support/manutils.py. you were hesitant in doing that for other parts of the setup.py, so what do you think? i feel it would be messy to include it all in setup.py...

license is acceptable: https://raw.githubusercontent.com/oz123/man-utils/master/LICENSE

@anarcat the project was a response to a bug report in python.org. It was made, and never really used unfortunately. I used it in pwman3 like you already saw. The usage is really straight forward. If you would like I can help directly via IRC or mail. 

importing in setup.py is dangerous as stuff is not installed yet when it runs.

@ThomasWaldmann i know about that specific concern, but i was able to workaround that fairly reliably in monkeysign in the past... are there other concerns?

I'ld prefer the man page being generated by a developer and commited to the repo than doing complex stuff triggered by setup.py (we just had a lot of troubles and pain with exactly that).

sure, but that command needs to be implemented somewhere. @oz123 proposed a solution that is standalone but that can also be integrated in `setup.py` (without necessarily being triggered in the default build, same as our current build_usage).

if it is not triggered at install/build time and does not import stuff except when actually using it, I guess we could use it.

note: debian has added a manpage by just using "make man" with the sphinx Makefile. It contains the same as our html docs, just in man page format.

@TW this is what #409 was trying to do as well.

As (IIRC) more than 1 linux dist is now generating their "own" man page by just using "make man", I think we should do that upstream, so they don't have to.

Building borgbackup for CentOS (#105) is at the moment not possible, cause the needed packages and dependencies for spinx are not available in epel. python3-spinx and python3-spinx_rtd_theme have a lot of deps.

@bpereto how about building a package without (html or manpage) docs? borg has builtin help and the html docs are on our web site.

@ThomasWaldmann 
found python2-sphinx exists in epel, so I can stick with that. and after fixing the unicode string literal, it worked.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,7 +1,10 @@
 # -*- encoding: utf-8 *-*
 import os
+import io
 import re
 import sys
+from collections import OrderedDict
+from datetime import datetime
 from glob import glob
 
 from distutils.command.build import build
@@ -326,6 +329,217 @@ def shipout(text):
         shipout(text)
 
 
+class build_man(Command):
+    description = 'build man pages'
+
+    user_options = []
+
+    see_also = {
+        'create': ('delete', 'prune', 'check', 'patterns', 'placeholders', 'compression'),
+        'recreate': ('patterns', 'placeholders', 'compression'),
+        'list': ('info', 'diff', 'prune', 'patterns'),
+        'info': ('list', 'diff'),
+        'init': ('create', 'delete', 'check', 'list', 'key-import', 'key-export', 'key-change-passphrase'),
+        'key-import': ('key-export', ),
+        'key-export': ('key-import', ),
+        'mount': ('umount', 'extract'),  # Would be cooler if these two were on the same page
+        'umount': ('mount', ),
+        'extract': ('mount', ),
+    }
+
+    rst_prelude = textwrap.dedent("""
+    .. role:: ref(title)
+
+    .. |project_name| replace:: Borg
+    """)
+
+    def initialize_options(self):
+        pass
+
+    def finalize_options(self):
+        pass
+
+    def run(self):
+        print('building man pages (in docs/man)', file=sys.stderr)
+        os.makedirs('docs/man', exist_ok=True)
+        # allows us to build docs without the C modules fully loaded during help generation
+        from borg.archiver import Archiver
+        parser = Archiver(prog='borg').parser
+
+        self.generate_level('', parser, Archiver)
+        self.build_topic_pages(Archiver)
+
+    def generate_level(self, prefix, parser, Archiver):
+        is_subcommand = False
+        choices = {}
+        for action in parser._actions:
+            if action.choices is not None and 'SubParsersAction' in str(action.__class__):
+                is_subcommand = True
+                for cmd, parser in action.choices.items():
+                    choices[prefix + cmd] = parser
+        if prefix and not choices:
+            return
+
+        for command, parser in sorted(choices.items()):
+            if command.startswith('debug') or command == 'help':
+                continue
+
+            man_title = 'borg-' + command.replace(' ', '-')
+            print('building man page', man_title + '(1)', file=sys.stderr)
+
+            if self.generate_level(command + ' ', parser, Archiver):
+                continue
+
+            doc, write = self.new_doc()
+            self.write_man_header(write, man_title, parser.description)
+
+            self.write_heading(write, 'SYNOPSIS')
+            write('borg', command, end='')
+            self.write_usage(write, parser)
+            write('\n')
+
+            self.write_heading(write, 'DESCRIPTION')
+            write(parser.epilog)
+
+            self.write_heading(write, 'OPTIONS')
+            write('See `borg-common(1)` for common options of Borg commands.')
+            write()
+            self.write_options(write, parser)
+
+            self.write_examples(write, command)
+
+            self.write_see_also(write, man_title)
+
+            self.gen_man_page(man_title, doc.getvalue())
+
+        # Generate the borg-common(1) man page with the common options.
+        if 'create' in choices:
+            doc, write = self.new_doc()
+            man_title = 'borg-common'
+            self.write_man_header(write, man_title, 'Common options of Borg commands')
+
+            common_options = [group for group in choices['create']._action_groups if group.title == 'Common options'][0]
+
+            self.write_heading(write, 'SYNOPSIS')
+            self.write_options_group(write, common_options)
+            self.write_see_also(write, man_title)
+            self.gen_man_page(man_title, doc.getvalue())
+
+        return is_subcommand
+
+    def build_topic_pages(self, Archiver):
+        for topic, text in Archiver.helptext.items():
+            doc, write = self.new_doc()
+            man_title = 'borg-' + topic
+            print('building man page', man_title + '(1)', file=sys.stderr)
+
+            self.write_man_header(write, man_title, 'Details regarding ' + topic)
+            self.write_heading(write, 'DESCRIPTION')
+            write(text)
+            self.gen_man_page(man_title, doc.getvalue())
+
+    def new_doc(self):
+        doc = io.StringIO(self.rst_prelude)
+        doc.read()
+        write = self.printer(doc)
+        return doc, write
+
+    def printer(self, fd):
+        def write(*args, **kwargs):
+            print(*args, file=fd, **kwargs)
+        return write
+
+    def write_heading(self, write, header, char='-', double_sided=False):
+        write()
+        if double_sided:
+            write(char * len(header))
+        write(header)
+        write(char * len(header))
+        write()
+
+    def write_man_header(self, write, title, description):
+        self.write_heading(write, title, '=', double_sided=True)
+        self.write_heading(write, description, double_sided=True)
+        # man page metadata
+        write(':Author: The Borg Collective')
+        write(':Date:', datetime.utcnow().date().isoformat())
+        write(':Manual section: 1')
+        write(':Manual group: borg backup tool')
+        write()
+
+    def write_examples(self, write, command):
+        with open('docs/usage.rst') as fd:
+            usage = fd.read()
+            usage_include = '.. include:: usage/%s.rst.inc' % command
+            begin = usage.find(usage_include)
+            end = usage.find('.. include', begin + 1)
+            examples = usage[begin:end]
+            examples = examples.replace(usage_include, '')
+            examples = examples.replace('Examples\n~~~~~~~~', '')
+            examples = examples.replace('Miscellaneous Help\n------------------', '')
+            examples = re.sub('^(~+)$', lambda matches: '+' * len(matches.group(0)), examples, flags=re.MULTILINE)
+            examples = examples.strip()
+        if examples:
+            self.write_heading(write, 'EXAMPLES', '-')
+            write(examples)
+
+    def write_see_also(self, write, man_title):
+        see_also = self.see_also.get(man_title.replace('borg-', ''), ())
+        see_also = ['`borg-%s(1)`' % s for s in see_also]
+        see_also.insert(0, '`borg-common(1)`')
+        self.write_heading(write, 'SEE ALSO')
+        write(', '.join(see_also))
+
+    def gen_man_page(self, name, rst):
+        from docutils.writers import manpage
+        from docutils.core import publish_string
+        man_page = publish_string(source=rst, writer=manpage.Writer())
+        with open('docs/man/%s.1' % name, 'wb') as fd:
+            fd.write(man_page)
+
+    def write_usage(self, write, parser):
+        if any(len(o.option_strings) for o in parser._actions):
+            write(' <options> ', end='')
+        for option in parser._actions:
+            if option.option_strings:
+                continue
+            write(option.metavar, end=' ')
+
+    def write_options(self, write, parser):
+        for group in parser._action_groups:
+            if group.title == 'Common options' or not group._group_actions:
+                continue
+            title = 'arguments' if group.title == 'positional arguments' else group.title
+            self.write_heading(write, title, '+')
+            self.write_options_group(write, group)
+
+    def write_options_group(self, write, group):
+        def is_positional_group(group):
+            return any(not o.option_strings for o in group._group_actions)
+
+        if is_positional_group(group):
+            for option in group._group_actions:
+                write(option.metavar)
+                write(textwrap.indent(option.help or '', ' ' * 4))
+            return
+
+        opts = OrderedDict()
+
+        for option in group._group_actions:
+            if option.metavar:
+                option_fmt = '%s ' + option.metavar
+            else:
+                option_fmt = '%s'
+            option_str = ', '.join(option_fmt % s for s in option.option_strings)
+            option_desc = textwrap.dedent((option.help or '') % option.__dict__)
+            opts[option_str] = textwrap.indent(option_desc, ' ' * 4)
+
+        padding = len(max(opts)) + 1
+
+        for option, desc in opts.items():
+            write(option.ljust(padding), desc)
+
+
 class build_api(Command):
     description = "generate a basic api.rst file based on the modules available"
 
@@ -361,6 +575,7 @@ def run(self):
     'build_ext': build_ext,
     'build_api': build_api,
     'build_usage': build_usage,
+    'build_man': build_man,
     'sdist': Sdist
 }
 
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -179,8 +179,7 @@ def build_matcher(excludes, paths):
         return matcher, include_patterns
 
     def do_serve(self, args):
-        """Start in server mode. This command is usually not used manually.
-        """
+        """Start in server mode. This command is usually not used manually."""
         return RepositoryServer(restrict_to_paths=args.restrict_to_paths, append_only=args.append_only).serve()
 
     @with_repository(create=True, exclusive=True, manifest=False)
@@ -2024,16 +2023,17 @@ def build_parser(self, prog=None):
                                    help='add a comment text to the archive')
         archive_group.add_argument('--timestamp', dest='timestamp',
                                    type=timestamp, default=None,
-                                   metavar='yyyy-mm-ddThh:mm:ss',
-                                   help='manually specify the archive creation date/time (UTC). '
+                                   metavar='TIMESTAMP',
+                                   help='manually specify the archive creation date/time (UTC, yyyy-mm-ddThh:mm:ss format). '
                                         'alternatively, give a reference file/directory.')
         archive_group.add_argument('-c', '--checkpoint-interval', dest='checkpoint_interval',
                                    type=int, default=1800, metavar='SECONDS',
                                    help='write checkpoint every SECONDS seconds (Default: 1800)')
         archive_group.add_argument('--chunker-params', dest='chunker_params',
                                    type=ChunkerParams, default=CHUNKER_PARAMS,
-                                   metavar='CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,HASH_WINDOW_SIZE',
-                                   help='specify the chunker parameters. default: %d,%d,%d,%d' % CHUNKER_PARAMS)
+                                   metavar='PARAMS',
+                                   help='specify the chunker parameters (CHUNK_MIN_EXP, CHUNK_MAX_EXP, '
+                                        'HASH_MASK_BITS, HASH_WINDOW_SIZE). default: %d,%d,%d,%d' % CHUNKER_PARAMS)
         archive_group.add_argument('-C', '--compression', dest='compression',
                                    type=CompressionSpec, default=dict(name='none'), metavar='COMPRESSION',
                                    help='select compression algorithm, see the output of the '
@@ -2204,12 +2204,15 @@ def build_parser(self, prog=None):
         See the "borg help patterns" command for more help on exclude patterns.
 
         The following keys are available for --format:
+
         """) + BaseFormatter.keys_help() + textwrap.dedent("""
 
-        -- Keys for listing repository archives:
+        Keys for listing repository archives:
+
         """) + ArchiveFormatter.keys_help() + textwrap.dedent("""
 
-        -- Keys for listing archive files:
+        Keys for listing archive files:
+
         """) + ItemFormatter.keys_help()
         subparser = subparsers.add_parser('list', parents=[common_parser], add_help=False,
                                           description=self.do_list.__doc__,
@@ -2348,7 +2351,7 @@ def build_parser(self, prog=None):
         Also, prune automatically removes checkpoint archives (incomplete archives left
         behind by interrupted backup runs) except if the checkpoint is the latest
         archive (and thus still needed). Checkpoint archives are not considered when
-        comparing archive counts against the retention limits (--keep-*).
+        comparing archive counts against the retention limits (--keep-X).
 
         If a prefix is set with -P, then only archives that start with the prefix are
         considered for deletion and only those archives count towards the totals
@@ -2607,8 +2610,8 @@ def build_parser(self, prog=None):
                                    help='add a comment text to the archive')
         archive_group.add_argument('--timestamp', dest='timestamp',
                                    type=timestamp, default=None,
-                                   metavar='yyyy-mm-ddThh:mm:ss',
-                                   help='manually specify the archive creation date/time (UTC). '
+                                   metavar='TIMESTAMP',
+                                   help='manually specify the archive creation date/time (UTC, yyyy-mm-ddThh:mm:ss format). '
                                         'alternatively, give a reference file/directory.')
         archive_group.add_argument('-C', '--compression', dest='compression',
                                    type=CompressionSpec, default=None, metavar='COMPRESSION',
@@ -2623,9 +2626,11 @@ def build_parser(self, prog=None):
                                    help='read compression patterns from COMPRESSIONCONFIG, see the output of the '
                                         '"borg help compression" command for details.')
         archive_group.add_argument('--chunker-params', dest='chunker_params',
-                                   type=ChunkerParams, default=None,
-                                   metavar='CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,HASH_WINDOW_SIZE',
-                                   help='specify the chunker parameters (or "default").')
+                                   type=ChunkerParams, default=CHUNKER_PARAMS,
+                                   metavar='PARAMS',
+                                   help='specify the chunker parameters (CHUNK_MIN_EXP, CHUNK_MAX_EXP, '
+                                        'HASH_MASK_BITS, HASH_WINDOW_SIZE) or "default" to use the current defaults. '
+                                        'default: %d,%d,%d,%d' % CHUNKER_PARAMS)
 
         subparser.add_argument('location', metavar='REPOSITORY_OR_ARCHIVE', nargs='?', default='',
                                type=location_validator(),


Test patch: