Package: markdown-it-pyrs
markdown-it-pyrsCurrently in Beta, feedback welcome!A Python interface formarkdown-it.rs(andplugins), using Rust for blazingly fast Markdown parsing ⚡️The goal of this project is to provide a fast, safe, extensible, and easy-to-use Markdown parser for Python.
It is complimentary tomarkdown-it-py, which is a pure Python implementation of markdown-it, and here we aim to follow as close as possible the API for that package.If you care primarily about speed, this is the library for you.
For example, benchmarking the two libraries when parsing the CommonMark Spec file, markdown-it-pyrs is20x faster than markdown-it-py.Name (time, ms)MinMaxMeanRoundsmarkdown-it-pyrs5.2177.9695.96885markdown-it-py122.696143.246131.4317The drawback is that the library vendors compiled Rust code, and so:Parser plugins cannot currently be written in Python and added dynamically to the parser.It can be more difficult to integrate into environments likepyiodideand py-script (but maybe not for long:https://discuss.python.org/t/support-wasm-wheels-on-pypi/21924/3).UsageFirst install the package:pipinstallmarkdown-it-pyrsThen use it like you wouldmarkdown-it-py:frommarkdown_it_pyrsimportMarkdownItmd=MarkdownIt("commonmark").enable("table")md.render("# Hello, world!")# '<h1>Hello, world!</h1>\n'markdown-it.rsdoes not generate a token stream, but instead directly generates aNodetree.
This is similar to themarkdown-it-py'sSyntaxTreeNodeclass, although the API is not identical.
(source mapping is also provided by byte-offset, rather than line only)md=(MarkdownIt("commonmark").enable("table").enable_many(["linkify","strikethrough"]))node=md.tree("# Hello, world!")print(node.walk())# [Node(root), Node(heading), Node(text)]print(node.pretty(srcmap=True,meta=True))# <root srcmap="0:15">#   <heading srcmap="0:15">#     level: 1#     <text srcmap="2:15">#       content: Hello, world!Note:Attributes of theNodeclass, such asNode.attrs, return acopyof the underlying data, and so mutating it will not affect what is stored on the node, e.g.frommarkdown_it_pyrsimportNodenode=Node("name")# don't do this!node.attrs["key"]="value"print(node.attrs)# {}# do this instead (Python 3.9+)node.attrs=node.attrs|{"key":"value"}print(node.attrs)# {"key": "value"}# Node.children is only a shallow copy though, so this is finechild=Node("child")node.children=[child]node.children[0].name="other"print(child.name)# "other"Command Line InterfaceA CLI is also provided, which can be used like this:echo"# Hello, world!"|markdown-it-pyrshtml-# <h1>Hello, world!</h1>echo"# Hello, world!"|markdown-it-pyrsast-# <root>#   <heading>#     <text>Replace-with a filename to read from a file,
and seemarkdown-it-pyrs --helpfor more options,
including initial configuration and enabling plugins.Initial ConfigurationInitialisingMarkdownIt("zero")will not enable any plugins, and so you can add only the ones you need.UseMarkdownIt("commonmark")to enable all the CommonMark plugins.UseMarkdownIt("gfm")to enable all the CommonMark plugins, plus the GitHub Flavoured Markdown plugins.PluginsAll syntax rules inmarkdown-it.rsare implemented as plugins.
Plugins can be added to the parser by callingenableorenable_manywith the name of the plugin.
The following plugins are currently supported:CommonMark Blocks:blockquote: Block quotes with>code: Indented code blocksfence: Backtick code blocksheading:#ATX headingshr:---horizontal ruleslheading:---underline setext headingslist:*unordered lists and1.ordered listsparagraph: Paragraphsreference: Link reference definitions[id]: src "title"CommonMark Inlines:autolink:<http://example.com>backticks:`code`emphasis:_emphasis_,*emphasis*,**strong**,__strong__entity:&amp;escape: backslash escaping\image:![alt](src "title")link:[text](src "title"),[text][id],[text]newline: hard line breakshtml_block: HTML blockshtml_inline: HTML inlineGitHub Flavoured Markdown (https://github.github.com/gfm):table:| foo | bar |
| --- | --- |
| baz | bim |strikethrough:~~strikethrough~~tasklist:- [x] tasklist itemautolink_ext: Extended autolink detection with "bare URLs" likehttps://example.comandwww.example.comtagfilter: HTML tag filtering, e.g.<script>tags are removedOthers:sourcepos: Add source mapping to rendered HTML, looks like this:<stuff data-sourcepos="1:1-2:3">, i.e.line:col-line:colreplacements: Typographic replacements, like--to—smartquotes: Smart quotes, like"to“linkify: Automatically linkify URLs withhttps://crates.io/crates/linkify(note currently this only matches URLs with a scheme, e.g.https://example.com)heading_anchors: Add heading anchors, with defaults like GitHubfront_matter: YAML front matterfootnote: Pandoc-style footnotes (seehttps://pandoc.org/MANUAL.html#footnotes)deflist: Definition lists (seehttps://pandoc.org/MANUAL.html#definition-lists)DevelopmentI'm quite new to Rust, so if you see something that could be improved, issues and PRs are welcome!PyO3andMaturinare used to build the Python package, by wrappingmarkdown-it.rsin a Python module.pre-commitis used to run code formatting and linting checks, andtoxis used to run tests.TODOImprovements:Allow to override options:lang_prefix: Prefix for language classes on fenced code blocksquotes: Quote characters, for smart quotesAdd plugins: ...Allow options for plugins:heading anchorstasklist checkboxes to be disabledfootnotes with options to turn on/off inline/collect/backrefsOpen issue upstream:notext_joinrule (to join adjacenttextandtext_specialtokens)heading_anchorsplugin does not allow for e.g. GitHub format where non-uniqueness is resolved by appending-1,-2, etc, and also removal of image textCapture reference nodes in ASTdisable rulesbetter "cross-language" AST representationdiffering behaviour of linkify and normalize_url/commonmark_extras test failuresquote characters for smart-quotes and lang_prefix for fence
should both be variable at run-time? (currently they both must be compiled)fix docstring inexamples/ferris/block_rule.rs::FerrisBlockScanner::run,
which currently describes the JS API not the new rust oneCapture "piece-wise" source maps for nested content, e.g. for when the source is split over multiple lines and nested in another block (could get inline herehttps://github.com/rlidwka/markdown-it.rs/blob/6f906b38c8ffc3cc651e67b448b3655b7d0debb3/src/parser/inline/mod.rs#L115)easier way to getroot.extitems in core rules; it seems at present you have to swap memory and reswap at the end of the rule, see e.g. theInlineParserRuleallowtest_rules_at_lineto parse what the calling rule is, so that other rules can decide whether to interrupt based on the calling rule (in thecheckfunction), I think this would then allow behaviour similar to whataltdid (possibly needed for footnote definition parsing)In general though, where back-compatibility is not required, I agree withdjotgoal 7, i.e. that block elements should not be allowed to interrupt other block elements without a newlineThe possibility to return multiple (sequential) nodes from anInlineRule.run, e.g.((node1, length1), (node2, length2), ...)This would be similar to docutilsIn theNodewalk methods, allow the function to return something to indicate whether to continue walking the children of the nodeAlso in walk, parse the "path" to get to the current node, e.g. list of parent nodes, to allow for backtracingsimilar tohttps://github.com/syntax-tree/unist-util-visit-parents,https://github.com/syntax-tree/unist-util-visit, etcis there a way to use amatchstatement, to match a Node against aNodeValueimplementation? (rather than if/else forNode.cast)Rule priority as an integer (similar to RST transform priority)Currently can only specifybeforeorafteranother rule or all rulesCan feel a little unclear for plugins, when to use attrs and when to add fields to node value.Maintenance: