Package: mcpyrate
mcpyrateAdvanced macro expander and language lab for Python. The focus is on correctness, feature-completeness for serious macro-enabled work, and simplicity, in that order.We aim at developer-friendliness.mcpyrateyields correctcoveragefor macro-enabled code, reports errors as early as possible, and makes it easy to display the steps of any macro expansion - with syntax highlighting, use site filename, and source line numbers:Figure 1.mcpyratestepping throughletseqfrom the demos.mcpyratebuilds onmcpy, with a similar explicit and compact approach, but with a lot of new features. Some of our features are strongly inspired bymacropy, such as quasiquotes, macro arguments, and expansion tracing. Features original tomcpyrateinclude a universal bootstrapper, integrated REPL system (including an IPython extension) and support for chainable whole-module source and AST transformers, developed from the earlier prototypesimacropyandpydialect; plus multi-phase compilation (a.k.a.staging; inspired by Racket), and identifier macros.We usesemantic versioning.mcpyrateis almost-but-not-quite compatible withmcpy2.0.0, hence the initial release is 3.0.0. There are some differences in the named parameters the expander provides to the macro functions; for details, searchthe main user manualfordifferences to mcpy.Some hypertext features of this README, such as local links to detailed documentation, are not supported when viewed on PyPI;view on GitHubto have those work properly.Table of ContentsmcpyrateFirst exampleRunning the extra examples in the testsFeaturesDocumentationDifferences to other macro expanders for PythonDifferences tomacropyDifferences tomcpyInstall & uninstallFrom PyPIFrom sourceUnderstanding the implementationEmacs syntax highlightingInstall (for beginners in Emacs customization)Why macros?First examplemcpyrategives you macro-enabled Python with just two source files:# mymacros.py with your macro definitionsdefecho(expr,**kw):print('Echo')returnexpr# application.pyfrommymacrosimportmacros,echoecho[6*7]Or even with just one source file:# application.pyfrommcpyrate.multiphaseimportmacros,phasewithphase[1]:defecho(expr,**kw):print('Echo')returnexprfrom__self__importmacros,echoecho[6*7]To run either example,macropython -m application, ormacropython application.py.More examples can be found in thedemo/subfolder. To run the demos after installingmcpyrate, go to themcpyrateproject directory, and invoke them likemacropython demo/anaphoric_if.py.Running the extra examples in the testsThe tests contain even more usage examples, including advanced ones. See themcpyrate/test/subfolder.Tests must be run using themcpyratein the source tree (instead of any installed one), because they expect to live in the modulemcpyrate.test, but thetestsubfolder is not part of the installation. Thus, if themcpyratetop-level module name resolves to an installed copy, there won't be a module namedmcpyrate.test.To run with themcpyratein the source tree, replacemacropythonwithpython3 -m mcpyrate.repl.macropython. For example, to run a demo,python3 -m mcpyrate.repl.macropython demo/anaphoric_if.py, or to run a test,python3 -m mcpyrate.repl.macropython -m mcpyrate.test.test_compiler. Here the first-mgoes topython3, whereas the second one goes tomacropython.If you just want to run all tests,python3 runtests.py.FeaturesAgile development tools.Multi-phase compilation: Use macros also in the same module where they are defined.Universal bootstrapper:macropython. Import and use macros in your main program.Interactive console:macropython -i. Import, define and use macros in a console session.Embeddable à lacode.InteractiveConsole. Seemcpyrate.repl.console.MacroConsole.IPython extensionmcpyrate.repl.iconsole. Import, define and use macros in an IPython session.Seefull documentation of the REPL system.Run-time compiler access.Expand, compile and run macro-enabled code snippets on the fly.Accepts source code and AST inputs. (Use quasiquotes to conveniently create ASTs.)Dynamically created code snippets support all the same features as importing code from a source file on disk.Seefull documentation of the compiler. Examples can be found inmcpyrate/test/test_compiler.py.Testing and debugging.Statement coverage is correctly reported by tools such asCoverage.py.Macro expansion errors are reported at macro expansion time, with use site traceback.Debug outputwith a step-by-step expansion breakdown. See macromcpyrate.debug.step_expansion.Has both expr and block modes. Usestep_expansion[...]orwith step_expansionas appropriate.The output issyntax-highlighted, andline-numberedbased onlinenofields from the AST.Also names of macros currently bound in the expander are highlighted bystep_expansion.Line numbers are taken fromstatementAST nodes.The invisible nodesast.Moduleandast.Exprare shown, since especiallyast.Expris a common trap for the unwary.To step the expansion of a run-time AST value, see the macromcpyrate.metatools.stepr.Documentation.Manual expand-once. Seeexpander.visit_once; get theexpanderas a named argument of your macro. See also theexpand1sandexpand1rmacros inmcpyrate.metatools.Lightning speed.Bytecode caches (.pyc) are created and kept up-to-date. Saves macro expansion cost at startup for unchanged modules. Makesmcpyratefaston average.Beside a.pysource file itself, we look at any macro definition files
it imports macros from, recursively, in amake-like fashion.The mtime is the latest of those of the source file and its macro-dependencies,
considered recursively, so that if any macro definition anywhere in the
macro-dependency tree of a source file is changed, Python will treat that
source file as "changed", thus re-expanding and recompiling it (hence,
updating the corresponding.pyc).CAUTION:PEP 552 - Deterministic pycsis not supported; we support only the defaultmtimeinvalidation mode, at least for now.Quasiquotes, with advanced features.Hygienically interpolate both regular valuesand macro names.Delayed macro expansion inside quasiquoted code. User-controllable.Inverse quasiquote operator. See functionmcpyrate.quotes.unastify.Convert a quasiquoted AST back into a direct AST, typically for further processing before re-quoting it.Not an unquote; we have those too, but the purpose of unquotes is to interpolate values into quoted code. The inverse quasiquote, instead, undoes the quasiquote operation itself, after any unquotes have already been applied.Seefull documentation of the quasiquote system.Macro arguments.Opt-in. Declare by using the@parametricmacrodecorator on your macro function.Use brackets to invoke, e.g.macroname[arg0, ...][expr]. If no args, just leave that part out, e.g.macroname[expr].Themacroname[arg0, ...]syntax works inexpr,blockanddecoratormacro invocations in place of a baremacroname.The named parameterargsis a rawlistof the macro argument ASTs. Empty if no args were sent, or if the macro function is not parametric.Identifier (a.k.a. name) macros.Opt-in. Declare by using the@namemacrodecorator on your macro function.Can be used for creating magic variables that may only appear inside specific macro invocations.Dialects, i.e. whole-module source and AST transforms.ThinkRacket's#lang, but for Python.Define languages that use Python's surface syntax, but change the semantics; or plug in a per-module transpiler that (at import time) compiles source code from some other programming language into macro-enabled Python. Also an ASToptimizercould be defined as a dialect. Dialects can be chained.Sky's the limit, really. See thedialectsmodules inunpythonicfor example dialects.For debugging,from mcpyrate.debug import dialects, StepExpansion.If writing a full-module AST transformer that splices the whole module into a template, seemcpyrate.splicing.splice_dialect.Seefull documentation of the dialect system.Conveniences.Relative macro-imports (for code in packages), e.g.from .other import macros, kittify.The expander automatically fixes missingctxattributes in the AST, so you don't need to care about those in your macros.In most cases, the expander also fills in correct source location information automatically (for coverage reporting). If you're discarding nodes from the input, then you may have to beslightly carefuland useast.copy_locationappropriately.Several block macros can be invoked in the samewith(equivalent to nesting them, with leftmost outermost).AST visitor and transformerà lamacropy'sWalker, to easily context-manage state for subtrees, and collect items across the whole walk.Full documentation.ASTmarkers(pseudo-nodes) for communication in a set of co-operating macros (and with the expander).gensymto create a fresh, unused lexical identifier.unparseto convert an AST to the corresponding source code, optionally with syntax highlighting (for terminal output).dumpto look at an AST representation directly, with (mostly) PEP8-compliant indentation, optionally with syntax highlighting (node types, field names, bare values).DocumentationThe full documentation ofmcpyratelives in thedoc/subfolder. Some quick links:Main user manual- start hereUsing macrosWriting macros- starting with a short tour of useful modules inmcpyrate.Quasiquotes andmcpyrate.metatoolsREPL andmacropythonThemcpyratecompilerThe import algorithm- how macros, dialects and multi-phase compilation interact.Multi-phase compilation- how to use a macro in the same module where it is defined.Invoking the compiler at run timeAST walkersDialectsTroubleshootingWe aim at complete documentation.If you find something is missing, pleasefile an issue. (And if you already figured out the thing that was missing from the docs, a documentation PR is also welcome!)Differences to other macro expanders for Pythonmcpyrateisnotdrop-in compatible withmacropyormcpy. This section summarizes the differences.It should be emphasized that what follows is a minimal list of differences. In addition, we provide many advanced features not available in previous macro expanders for Python, such as dialects (full-module transforms), multi-phase compilation, and run-time compiler access (with dynamic module creation).Differences tomacropymcpyratehas no macro registry; each macro function is its own dispatcher. See thesyntaxnamed parameter.Inmcpyrate, macros expand outside-in (and only outside-in) by default.Noyield. To expand inside-out, recurse explicitly in your macro implementation at the point where you want to switch to inside-out processing. See themain user manual.Named parameters filled by expander(**kwin the macro definition) are almost totally different.Nogen_symparameter; use the functionmcpyrate.gensym. We use UUIDs, avoiding the need for a lexical scan.Macro argumentsPassed using brackets.Macro expansion error reportingRaise an exception as usual, don'tassert False.QuasiquotesQuasiquoted code isn't automatically macro-expanded.Hygienic quasiquoting works differently.No separatehq[]. Instead, we provideh[], which is ahygienic unquotethat captures a value or a macro.You can capture arbitrary expressions, not just identifiers.You can capture macros, too.The equivalents ofast_literalandnamehave additional features, and an operator to interpolate alistof ASTs as anast.Tuplehas been added (one use case is to splice in a variable number of macro arguments to a quasiquoted macro invocation).AST walkersSimilar in spirit toast.NodeTransformerandast.NodeVisitor, but with functionality equivalent to that ofmacropy.core.walkers.Walker. But it works differently.Particularly, use explicit recursion as inast.NodeTransformer; there is nostop.For thectxmechanism, seewithstateandgeneric_withstate(the latter relates to the former asgeneric_visitrelates tovisitinast.NodeTransformer).Differences tomcpyNamed parameters filled by expanderNoto_source; use the functionmcpyrate.unparse. You might wantunparse(tree, debug=True, color=True).Noexpand_macros; ask forexpanderinstead, and callexpander.visit.Install & uninstallFrom PyPIpipinstallmcpyratepossibly with--user, if your OS is a *nix, and you feel lucky enough to use the system Python. If not, activate your venv first; the--userflag is then not needed.From sourceClone the repo from GitHub. Then, navigate to it in a terminal, and:python-msetupinstallpossibly with--user, if your OS is a *nix, and you feel lucky enough to use the system Python. If not, activate your venv first; the--userflag is then not needed.To uninstall:pipuninstallmcpyratebut first, make sure you're not in a folder that has amcpyratesubfolder -pipwill think it got a folder name instead of a package name, and become confused.Understanding the implementationWe follow themcpyphilosophy that macro expanders aren't rocket science. SeeCONTRIBUTING.md.Emacs syntax highlightingThis Elisp snippet adds syntax highlighting for keywords specific tomcpyrateto your Emacs setup:(defunmy/mcpyrate-syntax-highlight-setup()"Set up additional syntax highlighting for `mcpyrate` in Python mode.";; adapted from code in dash.el(let((more-keywords'("macros""dialects""q""u""n""a""s""t""h"));; How to make Emacs recognize your magic variables. Only for the anaphoric if demo.;; A list, like `more-keywords`, even though in the example there is only one item.(magic-variables'("it")))(font-lock-add-keywords'python-mode`((,(concat"\\_<"(regexp-optmagic-variables'paren)"\\_>")1font-lock-variable-name-face))'append)(font-lock-add-keywords'python-mode`((,(concat"\\_<"(regexp-optmore-keywords'paren)"\\_>")1font-lock-keyword-face))'append)))(add-hook'python-mode-hook'my/mcpyrate-syntax-highlight-setup)Known issue: For some reason, during a given session, this takes effect only starting with the second Python file opened. The first Python file opened during a session shows with the default Python syntax highlighting. Probably something to do with the initialization order of font-lock and whicheverpython-modeis being used.Tested withanaconda-mode.Install (for beginners in Emacs customization)If you use theSpacemacskit, the snippet can be inserted into the functiondotspacemacs/user-config. (If you use the Emacs key bindings,M-m f e dto open your config file.) Here'smy spacemacs.dfor reference; the syntax highlight code is inprettify-symbols-config.el, and it's invoked from the functiondotspacemacs/user-configininit.el.In a basic Emacs setup, the snippet goes into the~/.emacsstartup file, or if you have an.emacs.d/directory, then into~/.emacs.d/init.el.Why macros?Despite theirfearsomereputation,syntacticmacros are a clean solution to certain classes of problems. Main use cases of macros fall into a few (not necessarily completely orthogonal) categories:Syntactic abstraction, to extract a pattern that cannot be extracted as a regular run-time function. Regular function definitions are a tool for extracting certain kinds of patterns; macros are another such tool. Both these tools aim at eliminating boilerplate, by allowing the definition of reusable abstractions.Macros can replace design patterns, especially patterns that work around a language's limitations. See Norvig'sclassic presentation on design patterns. For a concrete example, seeSeibel.Source code access. Any operation that needs to get a copy of the source code of an expression (or of a code block) as well as run that same code is a prime candidate for a macro. This is useful for implementing tooling for e.g. debug-logging and testing.Evaluation order manipulation. By editing code, macros can change the order in which it gets evaluated, as well as decide whether a particular expression or statement runs at all.As an example, macros allow properly abstractingdelay/forcein astrictlanguage.forceis just a regular function, butdelayneeds to be a macro. Seeour delayed evaluation demo.Language-level featuresinspired by other programming languages. For example,unpythonicprovides expression-local variables (let), automatic tail call optimization (TCO), autocurry, lazy functions, and multi-shot continuations.Asthe Racket guide notes, this is especially convenient for language-level features not approved by some other language designer. Macros allow users toextendthe language.Dialectstake that idea one step further.Compile-time validation. See e.g.Alexis King (2016): Simple, safe multimethods in Racket. Our sister projectunpythonicalso uses macros to perform some simple checks for whether certain helper constructs appear in a valid position, and when not, errors out at compile time.Embedded domain-specific languages (eDSLs).Hereembeddedmeans the DSL seamlessly integrates into the surrounding programming language (the host language). With embedded DSLs, there is no need to implement a whole new parser for the DSL, and many operations can be borrowed from the host language. Infix arithmetic notation and regular expressions are common examples of eDSLs that come embedded in many programming languages.(Note that a general-purpose programming language does not strictly need to provide infix arithmetic; many Lisps do not. Of course, a form of infix arithmetic can be added as a macro; here'sa very compact Racket solution(search the page for"more operators").)The embedded approach significantly decreases the effort needed to implement a DSL, thus making small DSLs an attractive solution for a class of design problems. A language construction kitcan be much more usefulthan how it may sound at first.Mobile code, as pioneered bymacropy. Shuttle code between domains, while still allowing it to be written together in a single code base.That said,macros are the 'nuclear option' of software development. Often a good strategy is to implement as much as regular functions as reasonably possible, and then a small macro on top, for the parts that would not be possible (or overly verbose, or overly complex, or just overly hacky) otherwise.Our delayed evaluation demois a small example of this strategy.More extensive examples are the macro-enabled test frameworkunpythonic.test.fixtures, and theletconstructs inunpythonic.syntax(though in that case the macros are rather complex, to integrate with Python's lexical scoping). If curious about the"overly hacky"remark, compare the implementations ofunpythonic.ambandunpythonic.syntax.forall- the macro version is much cleaner.For examples of borrowing language features, look atGraham,Python'swithin Clojure,unpythonic.syntax, and these creations from the Racket community[1][2][3]. But observe also that macros are not always needed for this:pattern matching,resumable exceptions, multiple dispatch[1][2].