Change:
at least that's how much I currently understand, since macros are really counterintuitive sometimes. I don't think this is supposed to happen. Even if it is, the exact rules used for macro matching should definitely be documented somewhere (I think there's not even an RFC).
It seems like the parser in particular is very greedy, or something. For example, eliminates the error (as does , but I assume this is reduced from something where you can't do that). However, fixing this (if it can be fixed) is probably a breaking change :(
Found the culprit: Do you really think this would break something? I actually don't think it would, since a fix for this should only accept more macro invocations.
On second thought... I think that's not the direct cause, since all other fragments behave roughly the same. Oh well, back to the drawing board
Okay, so as it turns out, all NT parsers introduce this bug (except , which you can't really test). In this particular case, changing from to only worked because idents and integral literals are both valid expressions. For example, this doesn't work: Now that this confusion is out of the way, I think I see what happens: When parsing as an expression, libsyntax panics because of a syntax error, which aborts compilation (the macro docs state that the parser fully commits to parsing such a nonterminal, so the fact that the third invocation doesn't work is expected behaviour). However, when parsing , it works. But since the token wasn't eaten, the second arm is tried (this is the bug!). It matches, of course, and the macro is accepted. So, fixing this would indeed be a breaking change, since the intended behaviour is (at least as far as I know), to reject the invocation, but this isn't happening.
I disagree that attempting the second arm is a bug. Ideally, should attempt each rule until it finds one that matches. From my perspective, the bug is that has no way to recover from failed parse attempts. Ideally, your last example should go something like this (using to represent cursor positions): - → → matched - → → input too long; next rule → → → matched - → syntax error; next rule. → → → matched
That would indeed be useful! But I think this comment implies that that isn't the intended behaviour: (stating that the macro parser fully commits to NTs implies to me that it doesn't backtrack to try other arms)
I believe that's referring to how it parses within a rule. Earley parsers can deal with ambiguities by tracking multiple potential parse forests (if I remember correctly; my understanding is a little vague). What it's saying is that it has to commit to parsing a non-terminal (i.e. higher-level productions like expressions) because the parser doesn't have any way to back out of a partial parse. So when it encounters one, it has to parse it, come hell or high water. Having the macro system not check successive rules once a rule starts matching would be apocalyptic: it would kill damn near every useful, non-trivial macro. We're talking mass hysteria, cats and dogs living together.
Fair enough. In that case the bug is just that the macro expander will panic when it can't parse an NT, so it can't backtrack. I also managed to dig up , which was closed as "not a bug", but this definitely feels like one.
Sorry to dig this up again, but isn't an a terminal, so your comments about NTs aren't applicable to this bug?
Correct! That's why I mentioned : // this could be handled like a token, since it is one (AFAIK, token == terminal)
I got bitten by this bug too. Note that I see two different issues here: The parser accepts , which according to is a bug. Indeed, according to the docs I would have expected the parser to refuse, but I can live with it either way and I can see where fixing the issue would cause code to break.The parser refuses , which should be a no-brainer: `` is not an ident, so the first rule should not even begin to match and no backtracking is required to move on to the next rule. Moreover I don't see where fixing this would break anything, since it would only allow code that currently does not compile (although I have no idea how hard fixing this may turn out to be).

Discussion:
return Error(span, "missing fragment specifier".to_string()); } } <del> TokenTree::MetaVarDecl(..) => { </del> <ins> TokenTree::MetaVarDecl(_, _, id) => { </ins> // Built-in nonterminals never start with these tokens, // so we can eliminate them from consideration. <del> match *token { token::CloseDelim(_) => {}, _ => bb_eis.push(ei), </del> <ins> if may_begin_with(&*id.name.as_str(), token) { bb_eis.push(ei); </ins> } } seq @ TokenTree::Delimited(..) | seq @ TokenTree::Token(_, DocComment(..)) => {

} } <ins> /// Checks whether a non-terminal may begin with a particular token. /// /// Returning `false` is a *stability guarantee* that such a matcher will *never* begin with that /// token. Be conservative (return true) if not sure. fn may_begin_with(name: &str, token: &Token) -> bool { /// Checks whether the non-terminal may contain a single (non-keyword) identifier. fn may_be_ident(nt: &token::Nonterminal) -> bool { match *nt { token::NtItem(_) | token::NtBlock(_) | token::NtVis(_) => false, _ => true, } } match name { "expr" => token.can_begin_expr(), "ty" => token.can_begin_type(), "ident" => token.is_ident(), "vis" => match *token { // The follow-set of :vis + "priv" keyword + interpolated Token::Comma | Token::Ident(_) | Token::Interpolated(_) => true, _ => token.can_begin_type(), }, "block" => match *token { Token::OpenDelim(token::Brace) => true, Token::Interpolated(ref nt) => match nt.0 { token::NtItem(_) | token::NtPat(_) | token::NtTy(_) | token::NtIdent(_) | token::NtMeta(_) | token::NtPath(_) | token::NtVis(_) => false, // none of these may start with '{'. _ => true, }, _ => false, }, "path" | "meta" => match *token { Token::ModSep | Token::Ident(_) => true, Token::Interpolated(ref nt) => match nt.0 { token::NtPath(_) | token::NtMeta(_) => true, _ => may_be_ident(&nt.0), }, _ => false, }, "pat" => match *token { Token::Ident(_) |               // box, ref, mut, and other identifiers (can stricten) Token::OpenDelim(token::Paren) |    // tuple pattern Token::OpenDelim(token::Bracket) |  // slice pattern Token::BinOp(token::And) |          // reference Token::BinOp(token::Minus) |        // negative literal Token::AndAnd |                     // double reference Token::Literal(..) |                // literal Token::DotDot |                     // range pattern (future compat) Token::DotDotDot |                  // range pattern (future compat) Token::ModSep |                     // path Token::Lt |                         // path (UFCS constant) Token::BinOp(token::Shl) |          // path (double UFCS) Token::Underscore => true,          // placeholder Token::Interpolated(ref nt) => may_be_ident(&nt.0), _ => false, }, _ => match *token { token::CloseDelim(_) => false, _ => true, }, } } </ins> fn parse_nt<'a>(p: &mut Parser<'a>, sp: Span, name: &str) -> Nonterminal { if name == "tt" { return token::NtTT(p.parse_token_tree());

// except according to those terms. fn main() { <del> panic!(@); //~ ERROR expected expression, found `@` </del> <ins> panic!(@); //~ ERROR no rules expected the token `@` </ins> }

// except according to those terms. pub fn main() { <del> vec![,]; //~ ERROR expected expression, found `,` </del> <ins> vec![,]; //~ ERROR no rules expected the token `,` </ins> }

<ins> // Copyright 2017 The Rust Project Developers. See the COPYRIGHT // file at the top-level directory of this distribution and at // http://rust-lang.org/COPYRIGHT. // // Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or // http://www.apache.org/licenses/LICENSE-2.0> or the MIT license // <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your // option. This file may not be copied, modified, or distributed // except according to those terms. #![feature(macro_vis_matcher)] //{{{ issue 40569 ============================================================== macro_rules! my_struct { ($(#[$meta:meta])* $ident:ident) => { $(#[$meta])* struct $ident; } } my_struct!(#[derive(Debug, PartialEq)] Foo40569); fn test_40569() { assert_eq!(Foo40569, Foo40569); } //}}} //{{{ issue 26444 ============================================================== macro_rules! foo_26444 { ($($beginning:ident),*; $middle:ident; $($end:ident),*) => { stringify!($($beginning,)* $middle $(,$end)*) } } fn test_26444() { assert_eq!("a , b , c , d , e", foo_26444!(a, b; c; d, e)); assert_eq!("f", foo_26444!(; f ;)); } macro_rules! pat_26444 { ($fname:ident $($arg:pat)* =) => {} } pat_26444!(foo 1 2 5...7 =); pat_26444!(bar Some(ref x) Ok(ref mut y) &(w, z) =); //}}} //{{{ issue 40984 ============================================================== macro_rules! thread_local_40984 { () => {}; ($(#[$attr:meta])* $vis:vis static $name:ident: $t:ty = $init:expr; $($rest:tt)*) => { thread_local_40984!($($rest)*); }; ($(#[$attr:meta])* $vis:vis static $name:ident: $t:ty = $init:expr) => {}; } thread_local_40984! { // no docs #[allow(unused)] static FOO: i32 = 42; /// docs pub static BAR: String = String::from("bar"); // look at these restrictions!! pub(crate) static BAZ: usize = 0; pub(in foo) static QUUX: usize = 0; } //}}} //{{{ issue 35650 ============================================================== macro_rules! size { ($ty:ty) => { std::mem::size_of::<$ty>() }; ($size:tt) => { $size }; } fn test_35650() { assert_eq!(size!(u64), 8); assert_eq!(size!(5), 5); } //}}} //{{{ issue 27832 ============================================================== macro_rules! m { ( $i:ident ) => (); ( $t:tt $j:tt ) => (); } m!(c); m!(t 9); m!(0 9); m!(struct); m!(struct Foo); macro_rules! m2 { ( $b:expr ) => (); ( $t:tt $u:tt ) => (); } m2!(3); m2!(1 2); m2!(_ 1); m2!(enum Foo); //}}} //{{{ issue 39964 ============================================================== macro_rules! foo_39964 { ($a:ident) => {}; (_) => {}; } foo_39964!(_); //}}} //{{{ issue 34030 ============================================================== macro_rules! foo_34030 { ($($t:ident),* /) => {}; } foo_34030!(a, b/); foo_34030!(a/); foo_34030!(/); //}}} //{{{ issue 24189 ============================================================== macro_rules! foo_24189 { ( pub enum $name:ident { $( #[$attr:meta] )* $var:ident } ) => { pub enum $name { $( #[$attr] )* $var } }; } foo_24189! { pub enum Foo24189 { #[doc = "Bar"] Baz } } macro_rules! serializable { ( $(#[$struct_meta:meta])* pub struct $name:ident { $( $(#[$field_meta:meta])* $field:ident: $type_:ty ),* , } ) => { $(#[$struct_meta])* pub struct $name { $( $(#[$field_meta])* $field: $type_ ),* , } } } serializable! { #[allow(dead_code)] /// This is a test pub struct Tester { #[allow(dead_code)] name: String, } } macro_rules! foo_24189_c { ( $( > )* $x:ident ) => { }; } foo_24189_c!( > a ); fn test_24189() { let _ = Foo24189::Baz; let _ = Tester { name: "".to_owned() }; } //}}} //{{{ some more tests ========================================================== macro_rules! test_block { (< $($b:block)* >) => {} } test_block!(<>); test_block!(<{}>); test_block!(<{1}{2}>); macro_rules! test_ty { ($($t:ty),* $(,)*) => {} } test_ty!(); test_ty!(,); test_ty!(u8); test_ty!(u8,); macro_rules! test_path { ($($t:path),* $(,)*) => {} } test_path!(); test_path!(,); test_path!(::std); test_path!(std::u8,); test_path!(any, super, super::super::self::path, X<Y>::Z<'a, T=U>); macro_rules! test_meta_block { ($($m:meta)* $b:block) => {}; } test_meta_block!(windows {}); //}}} fn main() { test_26444(); test_40569(); test_35650(); test_24189(); } </ins>