mozilla/rust — pull request #11151 by sfackler (2013-12-26 16:32:09+00:00)
Title: RFC: Externally loadable syntax extensions

This is a first pass on support for procedural macros that aren't hardcoded into libsyntax. It is **not yet ready to merge** but I've opened a PR to have a chance to discuss some open questions and implementation issues.

Example
=======
Here's a silly example showing off the basics:

my_synext.rs
```rust
#[feature(managed_boxes, globs)];

extern mod syntax;

use syntax::ast::{Name, token_tree};
use syntax::codemap::Span;
use syntax::ext::base::*;
use syntax::parse::token;

#[no_mangle]
pub fn macro_crate_registration(register: |Name, SyntaxExtension|) {
register(token::intern(&"make_a_1"),
NormalTT(@SyntaxExpanderTT {
expander: SyntaxExpanderTTExpanderWithoutContext(f),
span: None,
} as @SyntaxExpanderTTTrait,
None));
}

pub fn expand_make_a_1(cx: @ExtCtxt, sp: Span, tts: &[token_tree]) -> MacResult {
if !tts.is_empty() {
cx.span_fatal(sp, "make_a_1 takes no arguments");
}
MRExpr(quote_expr!(cx, 1i))
}
```

main.rs:
```rust
macro_crate!(extern mod my_synext;)

fn main() {
assert_eq!(1, make_a_1!());
}
```

Overview
=======
Crates that contain syntax extensions need to define a function with the following signature:
```rust
#[no_mangle]
pub fn macro_crate_registration(register: |ast::Name, ext::base::SyntaxExtension|) { ... }
```
that should call the `register` closure with each extension it defines.

Crates that wish to use externally loadable syntax extensions load them via the `macro_crate!` macro. It wraps a single `extern mod` declaration, which is dynamically loaded by the compiler. All extensions registered by the specified crate are loaded with the same scoping rules as `macro_rules!` macros.

Open questions
===========
* Does the `macro_crate` syntax make sense? It wraps an entire `extern mod` declaration which looks a bit weird but is nice in the sense that the crate lookup logic can be identical between normal external crates and external macro crates. If the `extern mod` syntax, changes, this will get it for free, etc.
* Is the magic name `macro_crate_registration` the right way to handle extension registration? It could alternatively be handled by a function annotated with `#[macro_registration]` I guess.
* The crate loading logic lives inside of librustc, which means that the syntax extension infrastructure can't directly access it. I've worked around this by passing a `CrateLoader` trait object from the driver to libsyntax that can call back into the crate loading logic. It should be possible to pull things apart enough that this isn't necessary anymore, but it will be an enormous refactoring project. I think we'll need to create a couple of new libraries: libsynext libmetadata/ty and libmiddle.
* Item decorator extensions can be loaded but the `deriving` decorator itself can't be extended so you'd need to do e.g. `#[deriving_MyTrait] #[deriving(Clone)]` instead of `#[deriving(MyTrait, Clone)]`. Is this something worth bothering with for now?

Remaining work
===========
- [ ] There is not yet support for rustdoc downloading and compiling referenced macro crates as it does for other referenced crates. This shouldn't be too hard I think.
- [ ] This is not testable at stage1 and sketchily testable at stages above that. The stage *n* rustc links against the stage *n-1* libsyntax and librustc. Unfortunately, crates in the test/auxiliary directory link against the stage *n* libstd, libextra, libsyntax, etc. This causes macro crates to fail to properly dynamically link into rustc since names end up being mangled slightly differently. In addition, when rustc is actually installed onto a system, there are actually do copies of libsyntax, libstd, etc: the ones that user code links against and a separate set from the previous stage that rustc itself uses. By this point in the bootstrap process, the two library versions *should probably* be binary compatible, but it doesn't seem like a sure thing. Fixing this is apparently hard, but necessary to properly cross compile as well and is being tracked in #11145.
- [ ] `macro_rules!` style macros aren't being exported. Now that the crate loading infrastructure is there, this should just require serializing the AST of the macros into the crate metadata and yanking them out again, but I'm not very familiar with that part of the compiler.
- [ ] The `macro_crate_registration` function isn't type-checked when it's loaded. I poked around in the `csearch` infrastructure a bit but didn't find any super obvious ways of checking the type of an item with a certain name. Fixing this may also eliminate the need to `#[no_mangle]` the registration function.