Package: sciengdox
sciengdoxA python package for creating scientific and engineering documents
viapandocincluding inline-executable Python code.Key FeaturesPandoc filterfor converting pandoc
markdown to other formats (especially HTML and PDF).Codeblock executionHelper functions for generating tables, SVGmatplotlibplots, etc.MotivationThis is inspired bypweave,codebraid,knitr, and cousins, but I always seemed to have
to do some pre/post-processing to get things the way I want them.  I already use
other pandoc filters (e.g. pandoc-citeproc, pandoc-crossref), so why not simply
have another pandoc filter that will execute inline code and insert the results?Another key is getting quality diagrams from scientific python code.  For
example, pweave automatically inserts generated images, but there doesn't seem
to be a way to get SVG images without, again, pre- and post-processing in
another script.  SVG plots are, obviously, scalable and work much better for web
and PDF outputs.DevelopmentUsepoetryfor local environment management.
After cloning the repository:$cd<project-repo>
$poetryinstall
$poetryshellTo package and release:$poetrybuild
$poetrypublishBe sure toconfigure your credentialsprior to publishing.See alsothis page.Use and ExampleAn example Pandoc markdown file can be found inexample.  To process this
file, you need to havepandocinstalled and in your
path.  You also need to install the Pandoc filterspandoc-crossrefandpandoc-citeprocwhich provide nice
cross-referencing and reference/bibliography handling.InstallationWhen working with macOS or Linux orLinux on Windows via WSLpandocand the filters can be installed viaHomebrew.  (On
Linux/WSL, installlinuxbrew.)  Then
simply run:$brewinstallpandoc
$brewinstallpandoc-crossref
$brewinstallpandoc-citeproc
$brewinstalllibrsvgThen, of course, you need to install this filter and some other helpers for the
example.  The example helpers can be installed into your Python virtual
environment by running:$poetryinstall-EexamplesWindows-specific InstallTo set up an environment for Windows from scratch including terminals, editors,
Python, etc., seethis gist.
Additional installation steps to use this library include installingpandocand additional filters and utilities.Installpandocbydownloading the installerand following the standard instructions.  This should also get youpandoc-citeproc.exefor managing citations.Installpandoc-crossref(for managing intra-document cross-references) bydownloadingthe zipped
Windows release.  Unzip it, and movepandoc-crossref.exeto a location that is
on your system path.  For example, you can move to next topandoc-citeproc.exeinC:\Program Files\Pandoc.Finally, to handle embedding SVG images in PDF documents, this library relies onrsvg-convert.  This can be installed viaChocolatey.  Install the Chocolatey package manager
if you do not already have it, and then run:$chocoinstallrsvg-convertInstead of (or in addition to) Chocolately, you can also install theScoopinstaller.  Scoop does not currently have a formula
forrsvg-convert, but it can also be installed fromSourceForgeif you do not want to use Chocolatey.UTF-8 NoteThe underlying Pandoc filter for executing Python code embedded in your
documents relies on inter-process communication with a Python REPL behind the
scenes.  The default inter-process character encoding for Python on Windows isCP-1252, and this can cause
problems if your Python scripts generate output with special characters (and if
you are doing any scientific or engineering writing, they definitely will).Fortunately, this is easily worked-around by setting a Windows environment
variablePYTHONIOENCODINGtoutf-8.  After setting this, be sure to restart
any open terminal windows for the change to take effect.Matplotlib NoteIf you usematplotlibfor generating plots in inline Python code in your
document, you should explicity set theAggbackend early in your document (see
theexample/example.mdin this repo).  Without this, document conversion can
hang when thesvg_figurehelper function is called.Somewhere near the top of your Markdown document, add an executable Python code
block (without.echoso it won't appear in the output) that includes:importmatplotlibmatplotlib.use('Agg')Panflute Version NoteThis plugin relies on thepanflutePython package as a
bridge between Python andpandoc's Haskell.  ThepanfluteREADMElists API compatibility requirements between versions ofpanfluteand versions
ofpandoc.  Double-check this if you run into errors that mentionpanflutewhen compiling a document.If you are running an older version ofpandoc(e.g. 2.9.2) and start a new
project, you will need to explicitly install the compatiblepanfluteversion
in your environment with e.g.poetry add panflute@1.12.5.  Or
alternatively install apandocversion 2.11.x or later.PDF GenerationTo generate PDF files through Pandoc, you need to havexelatexinstalled.
On Linux/WSL:$sudoapt-getinstalltexlive-xetexOn macOS:$brewinstall--caskmactexOn Windows (without WSL):Download the MikTeX installerand install as
usual.  Then ensure that the binary folder is in your path (e.g.C:\Users\<username>\AppData\Local\Programs\MiKTeX 2.9\miktex\bin\x64\).  Note
that the first time you generate a document, MikTex will prompt you to install a
lot of packages, so watch for a MikTeX window popping up (possibly behind other
windows) and follow the prompts.FontsThe example templates rely on having a few fonts installed.
The fonts to get are the GoogleSource Sans Pro,Source Code Pro, andSource Serif Profamilies.On macOS or Windows (without WSL), these can simply be downloaded and installed
as you would any other font.  On Linux via WSL, you can install these normally
on the Windows side and then synchronize the Windows font folder to the Linux
side.  To do this, edit (usingsudo)/etc/fonts/local.confand add:<?xml version="1.0"?><!DOCTYPE fontconfig SYSTEM "fonts.dtd"><fontconfig><dir>/mnt/c/Windows/Fonts</dir></fontconfig>Then update the font cache on the Linux side:$sudofc-cache-fvStylesheetsThe example file uses an HTML template that includes a CSS stylesheet that is
generated fromSCSS.  To compile
this automatically, you need to haveSASS installed.On macOS, this can be installed via Homebrew:$brewinstallsass/sass/sassOn macOS/Linux/WSL/Windows it can be installed as a Node.js package (assuming
you already haveNode.js/npminstalled):$npminstall-gsassBuildingThis Python library provides a script,compiledoc, that will appear in yourpoetryorpipenvvirtual environment's path (or globally) once the library
is installed.  In general, you provide an output directory and an input markdown
file, and it will build an HTML output when the--htmlflag is used (and also
by default).$compiledoc-ooutput--htmlmydoc.mdTo build a PDF (viaxelatex):$compiledoc-ooutput--pdfmydoc.mdTo build a Markdown file with executable Python output included (e.g. for
debugging purposes), specify--md.  This will generate a file in the output
directory with (perhaps confusingly) the same name as the input:$compiledoc-ooutput--mdmydoc.mdTo build everything, specify--all:$compiledoc-ooutput--allmydoc.mdTo see all available command line options (for specifying templates, paths to
required external executables, static files like images and bibliography files,
etc.):$compiledoc--helpBuilding the ExampleOnce everything is setup, compile the example HTML file by running:$cdexample
$compiledoc-ooutputexample.mdOpenexample/output/example.htmlin your browser or use e.g. theLive
Serverplugin for VS Code.Auto RegenTo autoregenerate the document (e.g. the HTML version, the output of which is
watched by theLive Server), you can useWatchman.To create a trigger on a particular directory (doc/in this example) with anotebook.mdfile (change this to suit your purposes), copy the following into
a temporarytrigger.jsonfile:["trigger","doc/",{"name":"build_html","expression":["anyof",["match","notebook.md","wholename"]],"command":["poetry","run","compiledoc","-o","output","--html","notebook.md"]}]Then from your project root directory run:watchman-j<trigger.json
rmtrigger.jsonIt is also recommended that you add a.watchmanconfigfile to the watched
directory (e.g.doc/; also add.watchmanconfigto your.gitignore) with
the following contents:{"settle":3000}The settle parameter is in milliseconds.To turn off watchman:watchmanshutdown-serverTo turn it back on:cd<project-root>
watchmanwatchdoc/To watch the Watchman:tail-f/usr/local/var/run/watchman/<username>-state/log(Note that on Windows/WSL, to gettailto work the way you expect, you need to
add---disable-inotifyto the command; and yes, that's three-for some
reason.)Older pandoc VersionsForpandoc2.9 and earlier, the citation managerpandoc-citeprocwas a
separate filter that gets added to the compliation pipeline.  The path to this
filter can be specified on the command line tocompiledocwith the--pandoc-citeproc PATHflag.In newer versions ofpandoc(2.11 and beyond), the citeproc filter is built-in
to pandoc and is run by adding--citeprocto thepandoccommand-line.  Thecompiledocscript adds this by default unless the flag--use-pandoc-citeprocis added, in which case the older filter will be used.If you do not with to runciteprocat all, you can add the flagcompiledoc --no-citeprocto skip citation processing altogether.