Package: include-pandoc
include-pandocA wrapper for pandoc that processes include lines (like!include path/to/included/file) in markdown files before passing them to pandoc for further processing.It allows to assemble complex/modular texts with link texts referring to headers or link definitions contained in different files.It supports unlimited nested includes, and handles also relative include paths.Include-pandoc should work on any Linux version and distribution. It should work also on Windows and OSX but it is untested on such platforms yet.1. Installationsudo pip install include-pandoc && sudo include-pandoc --updateor (as user)pip install --user include-pandoc && include-pandoc --update2. Usageinclude-pandoc[PANDOC_OPTIONS][FILES]|[--update]Just callinclude-pandocinstead ofpandoc, with the same syntax you would use withpandoc. Without putting any files, standard input will be processed.If you update Pandoc on your system, before usinginclude-pandocagain, you have to runinclude-pandoc --update(otherwise you will get an error); this will update the wrapper with possible new Pandoc options implemented by the new installed version. If you installedpandoc-updateas root, you have to callinclude-pandoc --updateas root (f.e. with sudo).3. Include SyntaxIncludes must be onseparate lines, with theinclude statement at the very beginning of the line, withno quotesandno escaped characters. Some examples:!include relative_path/to/included/file.md!include /absolute_path/to/included/file.md!include path with spaces/to/included file.mdIf relative paths are used in includes, they are regarded as relative to the path of the file that contains the include line; also in nested/recursive includes, relative paths are regarded as relative to the path of the nested file that contains the include line, not to the main/parent file path.So ifmain.mdincludesparts/part1.mdand the latter in turn includesparts/subparts/subpart1.md:the include line inmain.mdwill be!include parts/part1.mdthe include line inpart1.mdwill be!include subparts/subpart1.md.4. Rationale4.1 The ProblemExisting solutions to implement includes in Pandoc are generally incomplete.There arefilters that handle only code block includes, likepandoc-include-code, that work well butonly for a specific purpose(i.e. including code within a markdown guide).There arefilterslikepandoc-includethat allow toinclude any kind of fileat any point of the markdown 'parent' file; however, they are alllimited by the architecture of Pandoc.
As explained inPandoc's guide:«A “filter” is a program that modifies the AST, between the reader and the writer:INPUT --reader--> AST --filter--> AST --writer--> OUTPUT.
Filters are “pipes” that read from standard input and write to standard output. They consume and produce a JSON representation of the pandoc AST».The problem is that links and link definitions are processed by Pandoc's reader, and filters come only after it, and they can process Pandoc's AST and not the 'original' text.As a consequence, when using Pandoc's include filters, if for instance a link text ([Super Goof]) contained in an included file (part1.md):references a header (# Super Goof) contained in another included file (part2.md),oris defined ([Super Goof]: #super_goof_id) in another included file (definitions.md),such link will remain 'unresolved' (i.e, it will be rendered as[Super Goof], without being changed to a link), because Pandoc processes links before the include filter merges all included files.This limitation prevents you to assemble working modular text documents through include filters in Pandoc.Alternative workarounds seem not viable, too.Some propose toconcatenate files when calling pandoc(pandoc 01.md 02.md 03.md), but it is a solution that may get very complicated to manage compared to an include based solution, especially when coming to complex modular texts:you would need to split the main document into small pieces, and further split -- or re-merge -- such pieces every time you want to move or change something, while it is much simpler to work on a main document that includes snippets from other files.Some propose touse m4 as a pre-processor, but this would lead to a lot of complications, too, because of the way m4 processes quotes and macros:for instance, if your text contains words corresponding to m4 macros likechangequoteordefine()or others, m4 processes them as macros, even if they are in the middle of a line (and in some cases, even if no parentheses follow the reserved word!); so you should avoid to use any m4 reserved words in your text, but this is not an acceptable limitation;moreover, you cannot use in your text, for any reason, the character sequences you set as begin-quote and end-quote delimiters for m4 - otherwise you may get m4 errors likedangling quote error-->m4:stdin:2: ERROR: end of file in string; so you should choose very long and unlikely sequences of characters as begin-quote and end-quote delimiters, that in turn may lead to very awkward include lines.4.2 The solutionI created a wrapper for Pandoc in order to process includes before passing the text(s) to Pandoc. It handles all the available options from the Pandoc version installed in your system and passes them transparently to Pandoc, but, before that, it processes the includes contained in the given file(s) (or in standard input if no filename is given). The chosen include syntax should not interfere with the remainder of the markdown text (like it happens instead with the m4 syntax).
In this way, you can create very complex modular texts with working links, no matter where the link is defined or where the referenced header is.