Package: manubot
Python utilities for Manubot: Manuscripts, open and automatedManubotis a workflow and set of tools for the next generation of scholarly publishing.
This repository contains a Python package with several Manubot-related utilities, as described in theusage sectionbelow.
Package documentation is available athttps://manubot.github.io/manubot(auto-generated from the Python source code).Themanubot citecommand-line interface retrieves and formats bibliographic metadata for user-supplied persistent identifiers like DOIs or PubMed IDs.
Themanubot processcommand-line interface prepares scholarly manuscripts for Pandoc consumption.
Themanubot processcommand is used by Manubot manuscripts, which are based off theRootstock template, to automate several aspects of manuscript generation.
Themanubot ai-revisioncommand is used to automatically revise a manuscript based on a set of AI-generated suggestions.
See Rootstock'smanuscript usage guidefor more information.Note:If you want to experience Manubot by editing an existing manuscript, seehttps://github.com/manubot/try-manubot.
If you want to create a new manuscript, seehttps://github.com/manubot/rootstock.To cite the Manubot project or for more information on its design and history, see:Open collaborative writing with ManubotDaniel S. Himmelstein, Vincent Rubinetti, David R. Slochower, Dongbo Hu, Venkat S. Malladi, Casey S. Greene, Anthony GitterPLOS Computational Biology(2019-06-24)https://doi.org/c7npDOI:10.1371/journal.pcbi.1007128· PMID:31233491· PMCID:PMC6611653The Manubot version of this manuscript is available athttps://greenelab.github.io/meta-review/.InstallationIf you are using themanubotPython package as part of a manuscript repository, installation of this package is handled though the Rootstock'senvironment specification.
For other use cases, this package can be installed viapip.Install the latest release versionfrom PyPI:pipinstall--upgrademanubotOr install from the source code onGitHub, using the version specified by a commit hash:COMMIT=d2160151e52750895571079a6e257beb6e0b1278
pipinstall--upgradegit+https://github.com/manubot/manubot@$COMMITThe--upgradeargument ensurespipupdates an existingmanubotinstallation if present.Some functions in this package requirePandoc,
which must beinstalledseparately on the system.
The pandoc-manubot-cite filter depends on Pandoc as well as panflute (a Python package).
Users must install acompatible version of panflutebased on their Pandoc version.
For example, on a system with Pandoc 2.9,
install the appropriate panflute likepip install panflute==1.12.5.UsageInstalling the python package creates themanubotcommand line program.
Here is the usage information as permanubot --help:usage: manubot [-h] [--version] {process,cite,webpage,ai-revision} ...

Manubot: the manuscript bot for scholarly writing

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

subcommands:
  All operations are done through subcommands:

  {process,cite,webpage,ai-revision}
    process             process manuscript content
    cite                citekey to CSL JSON command line utility
    webpage             deploy Manubot outputs to a webpage directory tree
    ai-revision         revise manuscript content with language modelsNote that all operations are done through the following sub-commands.ProcessThemanubot processprogram is the primary interface to using Manubot.
There are two required arguments:--content-directoryand--output-directory, which specify the respective paths to the content and output directories.
The content directory stores the manuscript source files.
Files generated by Manubot are saved to the output directory.One common setup is to create a directory for a manuscript that contains both thecontentandoutputdirectory.
Under this setup, you can run the Manubot using:manubotprocess\--skip-citations\--content-directory=content\--output-directory=outputSeemanubot process --helpfor documentation of all command line arguments:usage: manubot process [-h] --content-directory CONTENT_DIRECTORY
                       --output-directory OUTPUT_DIRECTORY
                       [--template-variables-path TEMPLATE_VARIABLES_PATH]
                       --skip-citations [--cache-directory CACHE_DIRECTORY]
                       [--clear-requests-cache] [--skip-remote]
                       [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]

Process manuscript content to create outputs for Pandoc consumption. Performs
bibliographic processing and templating.

options:
  -h, --help            show this help message and exit
  --content-directory CONTENT_DIRECTORY
                        Directory where manuscript content files are located.
  --output-directory OUTPUT_DIRECTORY
                        Directory to output files generated by this script.
  --template-variables-path TEMPLATE_VARIABLES_PATH
                        Path or URL of a file containing template variables
                        for jinja2. Serialization format is inferred from the
                        file extension, with support for JSON, YAML, and TOML.
                        If the format cannot be detected, the parser assumes
                        JSON. Specify this argument multiple times to read
                        multiple files. Variables can be applied to a
                        namespace (i.e. stored under a dictionary key) like
                        `--template-variables-path=namespace=path_or_url`.
                        Namespaces must match the regex `[a-zA-
                        Z_][a-zA-Z0-9_]*`.
  --skip-citations      Skip citation and reference processing. Support for
                        citation and reference processing has been moved from
                        `manubot process` to the pandoc-manubot-cite filter.
                        Therefore this argument is now required. If citation-
                        tags.tsv is found in content, these tags will be
                        inserted in the markdown output using the reference-
                        link syntax for citekey aliases. Appends
                        content/manual-references*.* paths to Pandoc's
                        metadata.bibliography field.
  --cache-directory CACHE_DIRECTORY
                        Custom cache directory. If not specified, caches to
                        output-directory.
  --clear-requests-cache
  --skip-remote         Do not add the rootstock repository to the local git
                        repository remotes.
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the logging level for stderr loggingManual referencesManubot has the ability to rely on user-provided reference metadata rather than generating it.manubot processsearches the content directory for files containing manually-provided reference metadata that match the globmanual-references*.*.
These files are stored in the Pandoc metadatabibliographyfield, such that they can be loaded bypandoc-manubot-cite.Citemanubot citeis a command line utility to produce bibliographic metadata for citation keys.
The utility either outputs metadata asCSL JSON itemsor produces formatted references if--render.Citation keys should be in the formatprefix:accession.
For example, the following example generates Markdown-formatted references for four persistent identifiers:manubotcite--format=markdown\doi:10.1098/rsif.2017.0387pubmed:29424689pmc:PMC5640425arxiv:1806.05726The followingterminal recordingdemonstrates the main features ofmanubot cite(for a slightly outdated version):Additional usage information is available frommanubot cite --help:usage: manubot cite [-h] [--output OUTPUT]
                    [--format {csljson,cslyaml,plain,markdown,docx,html,jats} | --yml | --txt | --md]
                    [--csl CSL] [--bibliography BIBLIOGRAPHY]
                    [--no-infer-prefix] [--allow-invalid-csl-data]
                    [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                    citekeys [citekeys ...]

Generate bibliographic metadata in CSL JSON format for one or more citation
keys. Optionally, render metadata into formatted references using Pandoc. Text
outputs are UTF-8 encoded.

positional arguments:
  citekeys              One or more (space separated) citation keys to
                        generate bibliographic metadata for.

options:
  -h, --help            show this help message and exit
  --output OUTPUT       Specify a file to write output, otherwise default to
                        stdout.
  --format {csljson,cslyaml,plain,markdown,docx,html,jats}
                        Format to use for output file. csljson and cslyaml
                        output the CSL data. All other choices render the
                        references using Pandoc. If not specified, attempt to
                        infer this from the --output filename extension.
                        Otherwise, default to csljson.
  --yml                 Short for --format=cslyaml.
  --txt                 Short for --format=plain.
  --md                  Short for --format=markdown.
  --csl CSL             URL or path with CSL XML style used to style
                        references (i.e. Pandoc's --csl option). Defaults to
                        Manubot's style.
  --bibliography BIBLIOGRAPHY
                        File to read manual reference metadata. Specify
                        multiple times to load multiple files. Similar to
                        pandoc --bibliography.
  --no-infer-prefix     Do not attempt to infer the prefix for citekeys
                        without a known prefix.
  --allow-invalid-csl-data
                        Allow CSL Items that do not conform to the JSON
                        Schema. Skips CSL pruning.
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the logging level for stderr loggingPandoc filterThis package creates thepandoc-manubot-citePandoc filter,
providing access to Manubot's cite-by-ID functionality from within a Pandoc workflow.Options are set via Pandoc metadata fieldslisted in the docs.usage: pandoc-manubot-cite [-h] [--input [INPUT]] [--output [OUTPUT]]
                           target_format

Pandoc filter for citation by persistent identifier. Filters are command-line
programs that read and write a JSON-encoded abstract syntax tree for Pandoc.
Unless you are debugging, run this filter as part of a pandoc command by
specifying --filter=pandoc-manubot-cite.

positional arguments:
  target_format      output format of the pandoc command, as per Pandoc's --to
                     option

options:
  -h, --help         show this help message and exit
  --input [INPUT]    path read JSON input (defaults to stdin)
  --output [OUTPUT]  path to write JSON output (defaults to stdout)Other Pandoc filters exist that do something similar:pandoc-url2cite,pandoc-url2cite-hs, &pwcite.
Currently,pandoc-manubot-citesupports the most types of persistent identifiers.
We're interested in creating as much compatibility as possible between these filters and their syntaxes.Manual referencesManual references are loaded from thereferencesandbibliographyPandoc metadata fields.
If a manual reference filename ends with.jsonor.yaml, it's assumed to contain CSL Data (i.e. Citation Style Language JSON).
Otherwise, the format is inferred from the extension and converted to CSL JSON using thepandoc-citeproc --bib2jsonutility.
The standard citation key for manual references is inferred from the CSL JSONidornotefield.
When no prefix is provided, such asdoi:,url:, orraw:, araw:prefix is automatically added.
If multiple manual reference files load metadata for the same standard citationid, precedence is assigned according to descending filename order.WebpageThemanubot webpagecommand populates awebpagedirectory with Manubot output files.usage: manubot webpage [-h] [--checkout [CHECKOUT]] [--version VERSION]
                       [--timestamp] [--no-ots-cache | --ots-cache OTS_CACHE]
                       [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]

Update the webpage directory tree with Manubot output files. This command
should be run from the root directory of a Manubot manuscript that follows the
Rootstock layout, containing `output` and `webpage` directories. HTML and PDF
outputs are copied to the webpage directory, which is structured as static
source files for website hosting.

options:
  -h, --help            show this help message and exit
  --checkout [CHECKOUT]
                        branch to checkout /v directory contents from. For
                        example, --checkout=upstream/gh-pages. --checkout is
                        equivalent to --checkout=gh-pages. If --checkout is
                        ommitted, no checkout is performed.
  --version VERSION     Used to create webpage/v/{version} directory.
                        Generally a commit hash, tag, or 'local'. When
                        omitted, version defaults to the commit hash on CI
                        builds and 'local' elsewhere.
  --timestamp           timestamp versioned manuscripts in webpage/v using
                        OpenTimestamps. Specify this flag to create timestamps
                        for the current HTML and PDF outputs and upgrade any
                        timestamps from past manuscript versions.
  --no-ots-cache        disable the timestamp cache.
  --ots-cache OTS_CACHE
                        location for the timestamp cache (default:
                        ci/cache/ots).
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the logging level for stderr loggingAI-assisted academic authoringThemanubot ai-revisioncommand uses large language models fromOpenAIto automatically revise a manuscript and suggest text improvements.usage: manubot ai-revision [-h] --content-directory CONTENT_DIRECTORY
                           [--model-type MODEL_TYPE]
                           [--model-kwargs key=value [key=value ...]]
                           [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]

Revise manuscript content using AI models to suggest text improvements.

options:
  -h, --help            show this help message and exit
  --content-directory CONTENT_DIRECTORY
                        Directory where manuscript content files are located.
  --model-type MODEL_TYPE
                        Model type used to revise the manuscript. Default is
                        GPT3CompletionModel. It can be any subclass of
                        manubot_ai_editor.models.ManuscriptRevisionModel
  --model-kwargs key=value [key=value ...]
                        Keyword arguments for the revision model (--model-
                        type), with format key=value.
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the logging level for stderr loggingThe usual call is:manubot ai-revision --content-directory content/The parameters--model-typeand--model-kwargsare used for debugging purposes.
For example, since the tool splits the text into paragraphs, you might want to see if paragraphs were detected correctly.
The tool incurs a cost when using the OpenAI API, so this could be important to check for text with complicated structure.manubot ai-revision \
  --content-directory content/ \
  --model-type DummyManuscriptRevisionModel \
  --model-kwargs add_paragraph_marks=trueDevelopmentEnvironmentCreate a development environment using:condacreate--namemanubot-dev--channelconda-forge\python=3.11pandoc=2.11.3.1
condaactivatemanubot-dev# assumes conda >= 4.4pipinstall--editable".[webpage,dev]"CommandsBelow are some common commands used for development.
They assume the working directory is set to the repository's root,
and the conda environment is activated.# run the test suitepytest# install pre-commit git hooks (once per local clone).# The pre-commit checks declared in .pre-commit-config.yaml will now# run on changed files during git commits.pre-commitinstall# run the pre-commit checks (required to pass CI)pre-commitrun--all-files# commit despite failing pre-commit checks (will fail CI)gitcommit--no-verify# regenerate the README codeblocks for --help messagespythonmanubot/tests/test_readme.py# generate the docsportrayas_html--overwrite--output_dir=docs# process the example testing manuscriptmanubotprocess\--content-directory=manubot/process/tests/manuscripts/example/content\--output-directory=manubot/process/tests/manuscripts/example/output\--skip-citations\--log-level=INFORelease instructionsThis section is only relevant for project maintainers.
GitHub Actionsdeploysreleases toPyPI.To create a new release, bump the__version__inmanubot/__init__.py.
Then, set theTAGandOLD_TAGenvironment variables:TAG=v$(pythonsetup.py--version)# fetch tags from the upstream remote# (assumes upstream is the manubot organization remote)gitfetch--tagsupstreammain# get previous release tag, can hardcode like OLD_TAG=v0.3.1OLD_TAG=$(gitdescribe--tags--abbrev=0)The following commands can help draft release notes:# check out a branch for a pull request as neededgitcheckout-b"release-$TAG"# create release notes file if it doesn't existtouch"release-notes/$TAG.md"# commit list since previous tagecho$'\n\nCommits\n-------\n'>>"release-notes/$TAG.md"gitlog--oneline--decorate=no--reverse$OLD_TAG..HEAD>>"release-notes/$TAG.md"# commit authors since previous tagecho$'\n\nCode authors\n------------\n'>>"release-notes/$TAG.md"gitlog$OLD_TAG..HEAD--format='%aN <%aE>'|sort--unique>>"release-notes/$TAG.md"After a commit with the above updates is part ofupstream:main,
for example after a PR is merged,
use theGitHub interfaceto create a release with the new "Tag version".
MonitorGitHub ActionsandPyPIfor successful deployment of the release.Goals & AcknowledgmentsOur goal is to create scholarly infrastructure that encourages open science and assists reproducibility.
Accordingly, we hope for the Manubot software and philosophy to be adopted widely, by both academic and commercial entities.
As such, Manubot is free/libre and open source software (seeLICENSE.md).We would like to thank the contributors and funders whose support makes this project possible.
Specifically, Manubot development has been financially supported by:theAlfred P. Sloan FoundationinGrant G-2018-11163to@dhimmel.theGordon & Betty Moore Foundation(@DDD-Moore) inGrant GBMF4552to@cgreene.