Package: libro-org
AboutA collection of tools Libro.org uses to produce its ebooks, including basic setup of ebooks, text processing, and build tools.Forked fromStandard Ebooks tools.Installing this toolset usingpipxmakes thelibrocommand line executable available. Its various commands are described below, or you can uselibro helpto list them.Project statusFeatureStatusadapt commands to SpanishWIPLibro.org skeletonWIPshell completionsTO-DOInstallationThe toolset requires Python >= 3.6.To install the toolset locally for development and debugging, seeInstallation for Developers.Ubuntu 20.04 (Trusty) users# Install some pre-flight dependencies.sudoaptinstall-ycalibredefault-jregitpython3-devpython3-pippython3-venv# Install pipx.python3-mpipinstall--userpipx
python3-mpipxensurepath# Install the toolset.pipxinstalllibro-orgOptional: Install shell completions# Install ZSH completions.sudoln-s$HOME/.local/pipx/venvs/standardebooks/lib/python3.*/site-packages/se/completions/zsh/_se/usr/share/zsh/vendor-completions/_se&&hash-rf&&compinit# Install Bash completions.sudoln-s$HOME/.local/pipx/venvs/standardebooks/lib/python3.*/site-packages/se/completions/bash/se/usr/share/bash-completion/completions/se# Install Fish completions.sudoln-s$HOME/.local/pipx/venvs/standardebooks/lib/python3.*/site-packages/se/completions/fish/se$HOME/.config/fish/completions/se.fishFedora users# Install some pre-flight dependencies.sudodnfinstallcalibregitjava-1.8.0-openjdkpython3-develvim# Install pipx.python3-mpipinstall--userpipx
python3-mpipxensurepath# Install the toolset.pipxinstalllibro-orgOptional: Install shell completions# Install ZSH completions.sudoln-s$HOME/.local/pipx/venvs/standardebooks/lib/python3.*/site-packages/se/completions/zsh/_se/usr/share/zsh/vendor-completions/_se&&hash-rf&&compinit# Install Bash completions.sudoln-s$HOME/.local/pipx/venvs/standardebooks/lib/python3.*/site-packages/se/completions/bash/se/usr/share/bash-completion/completions/se# Install Fish completions.sudoln-s$HOME/.local/pipx/venvs/standardebooks/lib/python3.*/site-packages/se/completions/fish/se$HOME/.config/fish/completions/se.fishmacOS users (up to macOS 10.15)These instructions were tested on macOS 10.12 to 10.16, on Intel macs.Install theHomebrew package manager. Or, if you already have it installed, make sure it’s up to date:brewupdateInstall dependencies:# Install some pre-flight dependencies.brewinstallcairocalibregitopenjdkpipxpython
pipxensurepath
sudoln-sfn/usr/local/opt/openjdk/libexec/openjdk.jdk/Library/Java/JavaVirtualMachines/openjdk.jdk# Install the toolset.pipxinstalllibro-org# Optional: Bash users who have set up bash-completion via brew can install tab completion.ln-s$HOME/.local/pipx/venvs/standardebooks/lib/python3.*/site-packages/se/completions/bash/se$(brew--prefix)/etc/bash_completion.d/se# Optional: Fish users can install tab completion.ln-s$HOME/.local/pipx/venvs/standardebooks/lib/python3.*/site-packages/se/completions/fish/se$HOME/.config/fish/completions/se.fishOpenBSD 6.6 UsersThese instructions were tested on OpenBSD 6.6, but may also work on the 6.5 release as well.Create a text file to feed intopkg_addcalled~/standard-ebooks-packages. It should contain the following:```shell
py3-pip--
py3-virtualenv--
py3-gitdb--
jdk--%11
calibre--
git--
vim--
```Optionally, replacevim--withvim--gtk3to include gvim for its Unicode editing features.Install dependencies usingdoas pkg_add -ivl ~/standard-ebooks-packages. Follow linking instructions provided bypkg_addto save keystrokes, unless you want to have multiple python versions and pip versions. In my case, I randoas ln -sf /usr/local/bin/pip3.7 /usr/local/bin/pip.Add~/.local/binto your path.Runpip install --user pipxIf you’re usingkshfrom base and have already added~/.local/bin, you can skippipx ensurepathbecause this step is forbashusers.The rest of the process is similar to that used on other platforms:# Install the toolset.pipxinstalllibro-orgInstallation for developersIf you want to work on the toolset source, it’s helpful to tellpipxto install the package in “editable” mode. This will allow you to edit the source of the package live and see changes immediately, without having to uninstall and re-install the package.To do that, follow the general installation instructions above; but instead of doingpipx install libro-org, do the following:gitclonehttps://github.com/Libro-Org/tools.git
pipxinstall--editable./toolsNow thelibrobinary is in your path, and any edits you make to source files in thetools/directory are immediately reflected when executing the binary.Running commands on the entire corpusAs a developer, it’s often useful to run anlibrocommand likelibro lintorlibro buildon the entire corpus for testing purposes. This can be very time-consuming in a regular invocation (likelibro lint /path/to/ebook/repos/*), because each argument is processed sequentially. Instead of waiting for a single invocation to process all of its arguments sequentially, useGNU Parallelto start multiple invocations in parallel, with each one processing a single argument. For example:# Slow, each argument is processed in sequenceselint/path/to/ebook/repos/*# Fast, multiple invocations each process a single argument in parallelexportCOLUMNS;parallel--keep-orderselint:::/path/to/ebook/repos/*The toolset tries to detect when it’s being invoked fromparallel, and it adjusts its output to accomodate.We exportCOLUMNSbecauselibro lintneeds to know the width of the terminal so that it can format its tabular output correctly. We pass the--keep-orderflag to output results in the order we passed them in, which is useful if comparing the results of multiple runs.Linting withpylintandmypyBefore we can usepylintormypyon the toolset source, we have to inject them into the venvpipxcreated for thelibro-orgpackage:pipxinjectlibro-orgpylintmypyThen make sure to call thepylintandmypybinaries thatpipxinstalled in thelibro-orgvenv,notany other globally-installed binaries:cd/path/to/tools/repo$HOME/.local/pipx/venvs/libro-org/bin/pylintseTesting withpytestSimilar topylint, thepytestcommand can be injected into the venvpipxcreated for thestandardebookspackage:pipxinjectlibro-orgpytestThe tests are executed by callingpytestfrom the top level or your tools repo:cd/path/to/tools/repo$HOME/.local/pipx/venvs/libro-org/bin/pytestAdding testsTests are added under thetestsdirectory. Most of the tests are based around the idea of having “golden” output files. Each command is run against a set of input files and then the resulting output files are compared against the resulting golden files. The test fails if the output files do not match the golden files. The data files can be found in thetests/datadirectory.A custom test flag--save-golden-fileshas been added to automatically update the the golden files for the tests (in anoutdirectory for the command).The usual test development process is:Updateinfiles with new test data and/or change the command implementation.Runpytestand see some tests fail.Runpytest --save-golden-filesand then diff the data directory to ensure that theoutfiles are as expected.Commit changes (including newoutcontents).Another custom test flag--save-new-draftis also available. This flag is used to update the book skeleton, generated by thelibro create-draftcommand, that is used as input for the other tests. Whenever the draft contents change (e.g. edits to thecore.cssfile) thetests/data/draftfiles should be updated by callingpytest --save-new-draft.Code styleIn general we follow a relaxed version ofPEP 8. In particular, we use tabs instead of spaces, and there is no line length limit.Always use theregexmodule instead of theremodule.Help wantedWe need volunteers to take the lead on the following goals:Add more test cases to the test framework.Figure out if it’s possible to install Bash/ZSH completions using setup.py,withoutroot; this may not be possible?Writing installation instructions for Bash and ZSH completions for MacOS.Currently we install the whole Calibre package, which is very big, but it’s only used to convert epub to azw3. Can we inline Calibre’s azw3 conversion code in the./vendor/directory, to avoid having to install the entire package as a big dependency?Tool descriptionslibro buildBuild an ebook from a Standard Ebook source directory.libro build-imagesBuild ebook cover and titlepage images in a Standard Ebook source directory and place the output in DIRECTORY/src/epub/images/.libro cleanPrettify and canonicalize individual XHTML, SVG, or CSS files, or all XHTML, SVG, or CSS files in a source directory. Note that this only prettifies the source code; it doesn’t perform typography changes.libro compare-versionsUse Firefox to render and compare XHTML files in an ebook repository. Run on a dirty repository to visually compare the repository’s dirty state with its clean state. If a file renders differently, place screenshots of the new, original, and diff (if available) renderings in the current working directory. A file called diff.html is created to allow for side-by-side comparisons of original and new files.libro create-draftCreate a skeleton of a new Standard Ebook.libro dec2romanConvert a decimal number to a Roman numeral.libro extract-ebookExtract an EPUB, MOBI, or AZW3 ebook into ./FILENAME.extracted/ or a target directory.libro find-mismatched-diacriticsFind words with mismatched diacritics in Standard Ebook source directories. For example,cafein one file andcaféin another.libro helpList available SE commands.libro hyphenateInsert soft hyphens at syllable breaks in an XHTML file.libro interactive-srUse Vim to perform an interactive search and replace on a list of files. Use y/n/a to confirm (y) or reject (n) a replacement, or to replace (a)ll.libro lintCheck for various Standard Ebooks style errors.libro make-url-safeMake a string URL-safe.libro modernize-spellingModernize spelling of some archaic words, and replace words that may be archaically compounded with a dash to a more modern spelling. For example, replaceash-traywithashtray.libro prepare-releaseCalculate work word count, insert release date if not yet set, and update modified date and revision number.libro print-manifestPrint the  element for the given Standard Ebooks source directory to standard output, for use in that directory’s content.opf.libro print-spinePrint the  element for the given Standard Ebooks source directory to standard output, for use in that directory’s content.opf.libro print-titlePrint the expected value for an XHTML file’s<title>element.libro print-tocBuild a table of contents for an SE source directory and print to stdout.libro recompose-epubRecompose a Standard Ebooks source directory into a single HTML5 file, and print to standard output.libro renumber-endnotesRenumber all endnotes and noterefs sequentially from the beginning.libro reorder-endnotesIncrement the specified endnote and all following endnotes by 1.libro roman2decConvert a Roman numeral to a decimal number.libro semanticateApply some scriptable semantics rules from the Standard Ebooks semantics manual to a Standard Ebook source directory.libro split-fileSplit an XHTML file into many files at all instances of<!--se:split-->, and include a header template for each file.libro titlecaseConvert a string to titlecase.libro typogrifyApply some scriptable typography rules from the Standard Ebooks typography manual to a Standard Ebook source directory.libro unicode-namesDisplay Unicode code points, descriptions, and links to more details for each character in a string. Useful for differentiating between different flavors of spaces, dashes, and invisible characters like word joiners.libro versionPrint the version number and exit.libro word-countCount the number of words in an HTML file and optionally categorize by length.libro xpathPrint the results of an xpath expression evaluated against a set of XHTML files. The default namespace is removed.What a Libro.org source directory looks likeMany of these tools act on Libro.org source directories. Such directories have a consistent minimal structure:.
|-images/
|--cover.jpg
|--cover.source.jpg
|--cover.svg
|--titlepage.svg
|-src/
|--epub/
|---css/
|----core.css
|----local.css
|----se.css
|---images/
|----cover.svg
|----titlepage.svg
|---text/
|----colophon.xhtml
|----imprint.xhtml
|----titlepage.xhtml
|----uncopyright.xhtml
|---content.opf
|---onix.xml
|---toc.xhtml
|--META-INF/
|---container.xml
|--mimetype
|-LICENSE.md./images/contains source images for the cover and titlepages, as well as ebook-specific source images. Source images should be in their maximum available resolution, then compressed and placed in./src/epub/images/for distribution../src/epub/contains the actual epub files.