Package: tldp
tldp - tools for publishing from TLDP sources=============================================.. image:: https://api.travis-ci.org/martin-a-brown/python-tldp.svg:target: https://github.com/tLDP/python-tldp.. image:: http://img.shields.io/badge/license-MIT-brightgreen.svg:target: http://opensource.org/licenses/MIT:alt: MIT licenseThis package was written for the Linux Documentation Project (TLDP) to helpwith management and publication automation of source documents.  The primaryinterface provided is a command-line tool caalled `ldptool`.  The canonicallocation of this software is:https://github.com/tLDP/python-tldp/The `ldptool` executable can:- crawl through any number of source collection directories- crawl through a single output collection- match the sources to the outputs (based on document stem name)- describe supported source formats (`--formats`)- describe the meaning of document status (`--statustypes`)- describe the collection by type and status (`--summary`)- list out individual document type and status (`--list`)- build the expected (non-configurable) set of outputs (`--build`)- build and publish the outputs (`--publish`)- produce runnable shell script to STDOUT (`--script`)The tools in this package process source documents in the `TLDP documentrepository <https://github.com/tLDP/LDP>`_ and generate the following set ofoutputs from each source document.- .pdf, PDF- .txt, text- -single.html, a one-page HTML document- .html, a multipage HTML document(We may add other output formats; an epub format is under consideration.)Supported input formats are:- Asciidoc- Linuxdoc- Docbook SGML 3.x (though deprecated, please no new submissions)- Docbook SGML 4.x- Docbook XML 4.x- Docbook XML 5.x (basic support, as of 2016-03-10)Example usages--------------If your attempts to run the below commands don't work or generate errors, seealso `Minimal configuration`_.Here are some example usages against a live checkout of the LDP sourcerepository and a local cache of the output tree:To see what work needs to be done, `ldptool --list`::$ ldptool  --listorphan    <unknown>            Bugzilla-Guidenew       DocBook XML 4.x      DocBook-Demystification-HOWTOstale     DocBook XML 4.x      Linux-Dictionarybroken    DocBook SGML 3.x/4.x PHP-Nuke-HOWTOstale     Linuxdoc             User-Group-HOWTOTo see publication status of each document:::$ ldptool --list all | head -n 3published Linuxdoc             3-Button-Mousepublished Linuxdoc             3D-Modellingpublished Linuxdoc             4mb-LaptopsTo get more information about the newer or missing files in a specificdocument:::$ ldptool --verbose --list Linux-Dictionarystale     DocBook XML 4.x      Linux-Dictionarydoctype <class 'tldp.doctypes.docbook4xml.Docbook4XML'>output dir /home/mabrown/tmp/en/Linux-Dictionarysource file /home/mabrown/vcs/LDP/LDP/guide/docbook/Linux-Dictionary/Linux-Dictionary.xmlnewer source /home/mabrown/vcs/LDP/LDP/guide/docbook/Linux-Dictionary/Contributors.xmlnewer source /home/mabrown/vcs/LDP/LDP/guide/docbook/Linux-Dictionary/D.xmlnewer source /home/mabrown/vcs/LDP/LDP/guide/docbook/Linux-Dictionary/J.xmlnewer source /home/mabrown/vcs/LDP/LDP/guide/docbook/Linux-Dictionary/O.xmlnewer source /home/mabrown/vcs/LDP/LDP/guide/docbook/Linux-Dictionary/S.xmlTo see what the entire source collection looks like, use `ldptool --summary`:::$ ldptool --summaryBy Status Type--------------source     503  3-Button-Mouse, 3D-Modelling, 4mb-Laptops, and 500 more ...output     503  3-Button-Mouse, 3D-Modelling, 4mb-Laptops, and 500 more ...published  503  3-Button-Mouse, 3D-Modelling, 4mb-Laptops, and 500 more ...stale        0orphan       0broken       1  HOWTO-INDEXnew          0By Document Type----------------Linuxdoc              226  3-Button-Mouse, 3D-Modelling, and 224 more ...Docbook4XML           130  8021X-HOWTO, abs-guide, and 128 more ...Docbook5XML             1  Assembly-HOWTODocbookSGML           146  ACP-Modem, and 145 more ...To build and publish a single document:::$ ldptool --publish DocBook-Demystification-HOWTO$ ldptool --publish ~/vcs/LDP/LDP/howto/docbook/Valgrind-HOWTO.xmlTo build and publish anything that is new or updated work:::$ ldptool --publish$ ldptool --publish workTo (re-)build and publish everything, regardless of state:::$ ldptool --publish allTo generate a specific output (into a --builddir):::$ ldptool --build DocBook-Demystification-HOWTOTo generate all outputs into a --builddir (should exist):::$ ldptool --builddir ~/tmp/scratch-directory/ --build allTo build new/updated work, but pass over a trouble-maker:::$ ldptool --build --skip HOWTO-INDEXTo loudly generate all outputs, except a trouble-maker:::$ ldptool --build all --loglevel debug --skip HOWTO-INDEXTo print out a shell script for building a specific document:::$ ldptool --script TransparentProxy$ ldptool --script ~/vcs/LDP/LDP/howto/docbook/Assembly-HOWTO.xmlLogging-------The `ldptool` utility is largely written to be interactive or a supervisedbatch process.  It uses STDERR as its logstream and sets the default loglevelat logging.ERROR.  At this log level, in `--script`, `--build` and `--publish`mode, it should report nothing to STDERR.  To increase progress verbosity,setting the loglevel to info (`--loglevel info`) may help with understandingwhat work the tool is performing.  If you need to collect diagnosticinformation for troubleshooting or bug reports, `ldptool` supports `--logleveldebug`.Configuration-------------The `ldptool` comes with support for reading its settings from thecommand-line, environment or a system and/or user-specified configurationfile.  If you want to generate a sample configuration file to edit and uselater, you can run:::ldptool --dump-cfg > my-ldptool.cfgldptool --configfile my-ldptool.cfg --listLDPTOOL_CONFIGFILE=/path/to/ldptool.cfg ldptool --listSource document identification------------------------------TLDP's source repository contains many separate directories containingdocuments (e.g. LDP/howto/docbook, LDP/howto/linuxdoc).  Each of thesedirectories may contain documents; to `ldptool` each of these is a`--sourcedir`.A source document (in a `--sourcedir`) can be a file or a directory.  Here aretwo examples.  The Assembly-HOWTO.xml is an entire document stored as a singlefile.  The directory BRIDGE-STP-HOWTO exists and contains its main document, afile named BRIDGE-STP-HOWTO.sgml.  In the case of a source document that is adirectory, the stem name of the primary document must match the name of thedirectory.::Assembly-HOWTO.xmlBRIDGE-STP-HOWTO/BRIDGE-STP-HOWTO/BRIDGE-STP-HOWTO.sgmlBRIDGE-STP-HOWTO/imagesBRIDGE-STP-HOWTO/images/hardware-setup.epsBRIDGE-STP-HOWTO/images/hardware-setup.pngBRIDGE-STP-HOWTO/images/old-hardware-setup.epsBRIDGE-STP-HOWTO/images/old-hardware-setup.pngEach document for a single run of `ldptool` can be uniquely identified by itsstem name.  In the above, the stems are `Assembly-HOWTO` and`BRIDGE-STP-HOWTO`.  It is an error to have two documents with the same stemname and the second discovered document will be ignored.There is a directory containing the output collection.  Each directory is namedby the stem name of the source document and contains the output formats foreach source document.  Here are the corresponding output directories for theabove two documents:::Assembly-HOWTO/Assembly-HOWTO/Assembly-HOWTO.htmlAssembly-HOWTO/Assembly-HOWTO.pdfAssembly-HOWTO/Assembly-HOWTO-single.htmlAssembly-HOWTO/Assembly-HOWTO.txtAssembly-HOWTO/index.htmlAssembly-HOWTO/mips.htmlAssembly-HOWTO/nasm.html... and more ...BRIDGE-STP-HOWTO/BRIDGE-STP-HOWTO/BRIDGE-STP-HOWTO.htmlBRIDGE-STP-HOWTO/BRIDGE-STP-HOWTO.pdfBRIDGE-STP-HOWTO/BRIDGE-STP-HOWTO-single.htmlBRIDGE-STP-HOWTO/BRIDGE-STP-HOWTO.txtBRIDGE-STP-HOWTO/imagesBRIDGE-STP-HOWTO/images/hardware-setup.epsBRIDGE-STP-HOWTO/images/hardware-setup.pngBRIDGE-STP-HOWTO/images/old-hardware-setup.epsBRIDGE-STP-HOWTO/images/old-hardware-setup.pngBRIDGE-STP-HOWTO/index.html... and more ...Minimal configuration---------------------The most important configuration parameters that `ldptool` takes are the setof source directories (in which to find documents) and the output directory,in which to create the resulting outputs.  It will not be able to run unlessit has at least one --sourcedir and an existing --pubdir directory.If you have an LDP checkout in your home directory, here's an example whichwould process all of the Linuxdoc HOWTO docs:::mkdir LDP-output-treeldptool --sourcedir $HOME/LDP/LDP/howto/linuxdoc --pubdir LDP-output-treeIf you would like to create a sample configuration file for use later (or forcopying into the system location, `/etc/ldptool/ldptool.ini`, you can generateyour own config file as follows:::ldptool > sample-ldptool.cfg \--sourcedir $HOME/LDP/LDP/faq/linuxdoc/ \--sourcedir $HOME/LDP/LDP/guide/linuxdoc/ \--sourcedir $HOME/LDP/LDP/howto/linuxdoc/ \--sourcedir $HOME/LDP/LDP/howto/docbook/ \--sourcedir $HOME/LDP/LDP/guide/docbook/ \--sourcedir $HOME/LDP/LDP/ref/docbook/ \--sourcedir $HOME/LDP/LDP/faq/docbook/ \--pubdir $HOME/LDP-output/ \--loglevel info \--dump-cfgThen, you can run the same configuration again with:::ldptool --configfile sample-ldptool.cfgThe `ldptool` program tries to locate all of the tools it needs to processdocuments.  Each source format requires a certain set of tools, for example, toprocess DocBook 4.x XML, `ldptool` needs the executables xmllint, xstlproc,html2text, fop and dblatex.  It also requires the XSL files for generating FO,chunked HTML and single-page HTML.  All of the items are configurable on thecommand-line or in the configuration file, but here's a sample config filestanza:::[ldptool-docbook4xml]xslchunk = /usr/share/xml/docbook/stylesheet/ldp/html/tldp-sections.xslxslsingle = /usr/share/xml/docbook/stylesheet/ldp/html/tldp-one-page.xslfop = /usr/bin/fopdblatex = /usr/bin/dblatexxsltproc = /usr/bin/xsltprochtml2text = /usr/bin/html2textxslprint = /usr/share/xml/docbook/stylesheet/ldp/fo/tldp-print.xslxmllint = /usr/bin/xmllintThe above stanza was generated by running `ldptool --dump-cfg` on an Ubuntu14.04 system which had all of the software dependencies installed.  If yourdistribution does not supply ldp-docbook-xsl, for example, you would need tofetch those files, put them someplace in the filesystem and adjust yourconfiguration file or command-line invocations accordingly.Software dependencies---------------------There are a large number of packages listed here in the dependency set.  Thisis because the supporting software for processing Linuxdoc and the variousDocBook formats is split across many upstream packages and repositories.The generated python packages (see below) do not include the explicitdependencies to allow the package manager (e.g. apt, zypper, dnf) to installthe dependencies.  This would be a nice improvement.Here are the dependencies needed for this tool to run:Ubuntu / Debian+++++++++++++++- linuxdoc-tools{,-text,-latex}- docbook{,-dsssl,-xsl,-utils}- htmldoc{,-common}- xsltproc- fop- sgml2x- opensp- openjade- ldp-docbook-xsl- ldp-docbook-dsssl- html2text- docbook5-xml- docbook-xsl-ns- jing- asciidoc- libxml2-utilsOpenSUSE++++++++- htmldoc- openjade- sgmltool- html2text- docbook{,5}-xsl-stylesheets- docbook-dsssl-stylesheets- docbook-utils-minimal- docbook-utils- jing- asciidoc- libxml2-tools- libxslt-toolsThere are a few additional data files that are needed, specifically, the TLDPXSL and DSSSL files that are used by the respective DocBook SGML (openjade) andDocBook XML (xsltproc) processing engines to generate the various outputs.On Debian-based systems, there are packages available from the distributorcalled ldp-docbook-{xsl,dsssl}.  There aren't any such packages for RPM (yet).Supported Python versions-------------------------This package was developed against Python-2.7.8 and Python-3.4.1 (onOpenSUSE).  It has been used on Python-2.7.6 (Ubuntu-14.04) and Python-3.4.2 and Python-2.7.9 (on Debian 8).Continuous Integration testing information and coverage can be reviewed at`this project's Travis CI page <https://travis-ci.org/martin-a-brown/python-tldp/>`_.Installation------------This is a pure-Python package, and you should be able to use your favoritePython tool to install it on your system.  The python-tldp package (`ldptool`)requires a large number of other packages, most of which are outside of thePython ecosystem.  There's room for improvement here, but here are a fewtidbits.Build an RPM::python setup.py sdist && rpmbuild -ta ./dist/python-tldp-${VERSION}.tar.gzThere's a generated file, `contrib/tldp.spec`, which makes a few changes to thesetuptools stock-generated specfile.  It adds the dependencies, marks theconfiguration file as %config(noreplace), adds a manpage and names the binarypackage `python-tldp`.Build a DEB::Check to see if the package is available from upstream.  It may be included inthe Debian repositories already::apt-cache search tldpThe quick and dirty way is as follows::python setup.py --command-packages=stdeb.command bdist_debBut, there is also a `debian` directory.  If you are working straight from thegit checkout, you should be able to generate an installable (unsigned) Debianpackage with::bash contrib/debian-release.sh -us -ucInstall using pip:Unknown.  Because the tool relies so heavily on system-installed non-Pythontools, I have not bothered to try installing the package using pip.  It shouldwork equivalently as well as running the program straight from a checkout.If you learn anything here or have suggestions, for me, please feel free tosend them along.Links-----* `Canonical python-tldp repository <https://github.com/tLDP/python-tldp>`_* `Source tree on GitHub <https://github.com/tLDP/LDP>`_* `Output documentation tree (sample) <http://www.tldp.org/>`_