Code change:
let source = pprust::token_to_string(self); parse_stream_from_source_str(FileName::MacroExpansion, source, sess, Some(span)) }); <ins> // During early phases of the compiler the AST could get modified // directly (e.g. attributes added or removed) and the internal cache // of tokens my not be invalidated or updated. Consequently if the // "lossless" token stream disagrees with our actual stringification // (which has historically been much more battle-tested) then we go // with the lossy stream anyway (losing span information). // // Note that the comparison isn't `==` here to avoid comparing spans, // but it *also* is a "probable" equality which is a pretty weird // definition. We mostly want to catch actual changes to the AST // like a `#[cfg]` being processed or some weird `macro_rules!` // expansion. // // What we *don't* want to catch is the fact that a user-defined // literal like `0xf` is stringified as `15`, causing the cached token // stream to not be literal `==` token-wise (ignoring spans) to the // token stream we got from stringification. // // Instead the "probably equal" check here is "does each token // recursively have the same discriminant?" We basically don't look at // the token values here and assume that such fine grained modifications // of token streams doesn't happen. </ins> if let Some(tokens) = tokens { <del> if tokens.eq_unspanned(&tokens_for_real) { </del> <ins> if tokens.probably_equal_for_proc_macro(&tokens_for_real) { </ins> return tokens } } return tokens_for_real } <ins> // See comments in `interpolated_to_tokenstream` for why we care about // *probably* equal here rather than actual equality pub fn probably_equal_for_proc_macro(&self, other: &Token) -> bool { if mem::discriminant(self) != mem::discriminant(other) { return false } match (self, other) { (&Eq, &Eq) | (&Lt, &Lt) | (&Le, &Le) | (&EqEq, &EqEq) | (&Ne, &Ne) | (&Ge, &Ge) | (&Gt, &Gt) | (&AndAnd, &AndAnd) | (&OrOr, &OrOr) | (&Not, &Not) | (&Tilde, &Tilde) | (&At, &At) | (&Dot, &Dot) | (&DotDot, &DotDot) | (&DotDotDot, &DotDotDot) | (&DotDotEq, &DotDotEq) | (&DotEq, &DotEq) | (&Comma, &Comma) | (&Semi, &Semi) | (&Colon, &Colon) | (&ModSep, &ModSep) | (&RArrow, &RArrow) | (&LArrow, &LArrow) | (&FatArrow, &FatArrow) | (&Pound, &Pound) | (&Dollar, &Dollar) | (&Question, &Question) | (&Whitespace, &Whitespace) | (&Comment, &Comment) | (&Eof, &Eof) => true, (&BinOp(a), &BinOp(b)) | (&BinOpEq(a), &BinOpEq(b)) => a == b, (&OpenDelim(a), &OpenDelim(b)) | (&CloseDelim(a), &CloseDelim(b)) => a == b, (&DocComment(a), &DocComment(b)) | (&Shebang(a), &Shebang(b)) => a == b, (&Lifetime(a), &Lifetime(b)) => a.name == b.name, (&Ident(a, b), &Ident(c, d)) => a.name == c.name && b == d, (&Literal(ref a, b), &Literal(ref c, d)) => { b == d && a.probably_equal_for_proc_macro(c) } (&Interpolated(_), &Interpolated(_)) => false, _ => panic!("forgot to add a token?"), } } </ins> } #[derive(Clone, RustcEncodable, RustcDecodable, Eq, Hash)]

Related discussion:
text: This is minimized from the dramatic forum thread . The following code is fine in nightly-2018-04-12 but breaks with nightly-2018-04-15. Oddly, lots of minor perturbations make it succeed. For example remove the two characters and it works. Or remove the single quote in the string and it works.
Alrighty-roo. As everything these days with hygiene, there's a couple of bugs here. With this one though I think only two! Everything has to do here with basically. AKA anything that runs through that falls off The usual reasons we think you fall off the "happy path" are due to things like inner attributes, , macro expansion, etc. None of that's present here! It should be the case that above can be losslessly converted to a based on what we've cached. Turns out though had a bug in it (surprise surprise!). There we're using to compare the resulting two token streams which eventually bottoms out in . As it turns out this isn't quite what we want! The implementation is structural, but the token stream we get when stringifying is slightly different than what we parsed. Here in the example you're using the literal string , but rustc stringifies this token as . Similarly if you replaced the string with something like it'll be stringified as . This means that the tokens are not literally byte-for-byte equal, which means we fall off the happy path! tl;dr the token stream of is not equivalent to the token stream . This causes us to fall off the happy path and used a stringified token stream. OK so in the case above we're falling off the happy path when we didn't expect to, but even still there's some weird bad hygiene behavior when we're not on the happy path. There's still something to fix! It turns out that there's also a regression due to (found via bisection). Before that PR this example would compile just fine, afterwards though it's hitting this error. can you help diagnose this and see what's going on? The reduced test case I have right now is: (note the inner attribute on to force using a non-cached token stream)
I've posted a fix for the "happy path" at will actually fix example above.