Change:
This is minimized from the dramatic forum thread . The following code is fine in nightly-2018-04-12 but breaks with nightly-2018-04-15. Oddly, lots of minor perturbations make it succeed. For example remove the two characters and it works. Or remove the single quote in the string and it works.
Alrighty-roo. As everything these days with hygiene, there's a couple of bugs here. With this one though I think only two! Everything has to do here with basically. AKA anything that runs through that falls off The usual reasons we think you fall off the "happy path" are due to things like inner attributes, , macro expansion, etc. None of that's present here! It should be the case that above can be losslessly converted to a based on what we've cached. Turns out though had a bug in it (surprise surprise!). There we're using to compare the resulting two token streams which eventually bottoms out in . As it turns out this isn't quite what we want! The implementation is structural, but the token stream we get when stringifying is slightly different than what we parsed. Here in the example you're using the literal string , but rustc stringifies this token as . Similarly if you replaced the string with something like it'll be stringified as . This means that the tokens are not literally byte-for-byte equal, which means we fall off the happy path! tl;dr the token stream of is not equivalent to the token stream . This causes us to fall off the happy path and used a stringified token stream. OK so in the case above we're falling off the happy path when we didn't expect to, but even still there's some weird bad hygiene behavior when we're not on the happy path. There's still something to fix! It turns out that there's also a regression due to (found via bisection). Before that PR this example would compile just fine, afterwards though it's hitting this error. can you help diagnose this and see what's going on? The reduced test case I have right now is: (note the inner attribute on to force using a non-cached token stream)
I've posted a fix for the "happy path" at will actually fix example above. That PR does not, however, fix the regression from which is still reproducible with the example I gisted.
Any idea why my repro case above is fixed if you remove the tokens? It seems like should fall off the happy path just as much.
If you remove the tokens we're still on the stringification path (aka not the happy path). For whatever reason though the hygiene bug goes away. That's likely caused from a subtelty introduced in (or at least so I'm hoping). I don't know much about , so I'm hoping will be able to look at this and know what's going on!
I don't know much about , so I'm hoping will be able to look at this and know what's going on! I'll investigate.
is the commit causing the regression. Further investigation is in progress.
An issue recently popped up on Rocket () that appears to be identical to this one and is not resolved in any recent nightly. It's possible that the root cause is , however.

Discussion:
use tokenstream; use std::{cmp, fmt}; <ins> use std::mem; </ins> use rustc_data_structures::sync::{Lrc, Lock}; #[derive(Clone, RustcEncodable, RustcDecodable, PartialEq, Eq, Hash, Debug, Copy)]

ByteStr(_) | ByteStrRaw(..) => "byte string" } } <ins> // See comments in `interpolated_to_tokenstream` for why we care about // *probably* equal here rather than actual equality fn probably_equal_for_proc_macro(&self, other: &Lit) -> bool { mem::discriminant(self) == mem::discriminant(other) } </ins> } pub(crate) fn ident_can_begin_expr(ident: ast::Ident, is_raw: bool) -> bool {

// stream they came from. Here we attempt to extract these // lossless token streams before we fall back to the // stringification. <del> // // During early phases of the compiler, though, the AST could // get modified directly (e.g. attributes added or removed) and // the internal cache of tokens my not be invalidated or // updated. Consequently if the "lossless" token stream // disagrees with our actuall stringification (which has // historically been much more battle-tested) then we go with // the lossy stream anyway (losing span information). </del> let mut tokens = None; match nt.0 {

let source = pprust::token_to_string(self); parse_stream_from_source_str(FileName::MacroExpansion, source, sess, Some(span)) }); <ins> // During early phases of the compiler the AST could get modified // directly (e.g. attributes added or removed) and the internal cache // of tokens my not be invalidated or updated. Consequently if the // "lossless" token stream disagrees with our actual stringification // (which has historically been much more battle-tested) then we go // with the lossy stream anyway (losing span information). // // Note that the comparison isn't `==` here to avoid comparing spans, // but it *also* is a "probable" equality which is a pretty weird // definition. We mostly want to catch actual changes to the AST // like a `#[cfg]` being processed or some weird `macro_rules!` // expansion. // // What we *don't* want to catch is the fact that a user-defined // literal like `0xf` is stringified as `15`, causing the cached token // stream to not be literal `==` token-wise (ignoring spans) to the // token stream we got from stringification. // // Instead the "probably equal" check here is "does each token // recursively have the same discriminant?" We basically don't look at // the token values here and assume that such fine grained modifications // of token streams doesn't happen. </ins> if let Some(tokens) = tokens { <del> if tokens.eq_unspanned(&tokens_for_real) { </del> <ins> if tokens.probably_equal_for_proc_macro(&tokens_for_real) { </ins> return tokens } } return tokens_for_real } <ins> // See comments in `interpolated_to_tokenstream` for why we care about // *probably* equal here rather than actual equality pub fn probably_equal_for_proc_macro(&self, other: &Token) -> bool { if mem::discriminant(self) != mem::discriminant(other) { return false } match (self, other) { (&Eq, &Eq) | (&Lt, &Lt) | (&Le, &Le) | (&EqEq, &EqEq) | (&Ne, &Ne) | (&Ge, &Ge) | (&Gt, &Gt) | (&AndAnd, &AndAnd) | (&OrOr, &OrOr) | (&Not, &Not) | (&Tilde, &Tilde) | (&At, &At) | (&Dot, &Dot) | (&DotDot, &DotDot) | (&DotDotDot, &DotDotDot) | (&DotDotEq, &DotDotEq) | (&DotEq, &DotEq) | (&Comma, &Comma) | (&Semi, &Semi) | (&Colon, &Colon) | (&ModSep, &ModSep) | (&RArrow, &RArrow) | (&LArrow, &LArrow) | (&FatArrow, &FatArrow) | (&Pound, &Pound) | (&Dollar, &Dollar) | (&Question, &Question) | (&Whitespace, &Whitespace) | (&Comment, &Comment) | (&Eof, &Eof) => true, (&BinOp(a), &BinOp(b)) | (&BinOpEq(a), &BinOpEq(b)) => a == b, (&OpenDelim(a), &OpenDelim(b)) | (&CloseDelim(a), &CloseDelim(b)) => a == b, (&DocComment(a), &DocComment(b)) | (&Shebang(a), &Shebang(b)) => a == b, (&Lifetime(a), &Lifetime(b)) => a.name == b.name, (&Ident(a, b), &Ident(c, d)) => a.name == c.name && b == d, (&Literal(ref a, b), &Literal(ref c, d)) => { b == d && a.probably_equal_for_proc_macro(c) } (&Interpolated(_), &Interpolated(_)) => false, _ => panic!("forgot to add a token?"), } } </ins> } #[derive(Clone, RustcEncodable, RustcDecodable, Eq, Hash)]

} } <ins> // See comments in `interpolated_to_tokenstream` for why we care about // *probably* equal here rather than actual equality // // This is otherwise the same as `eq_unspanned`, only recursing with a // different method. pub fn probably_equal_for_proc_macro(&self, other: &TokenTree) -> bool { match (self, other) { (&TokenTree::Token(_, ref tk), &TokenTree::Token(_, ref tk2)) => { tk.probably_equal_for_proc_macro(tk2) } (&TokenTree::Delimited(_, ref dl), &TokenTree::Delimited(_, ref dl2)) => { dl.delim == dl2.delim && dl.stream().probably_equal_for_proc_macro(&dl2.stream()) } (_, _) => false, } } </ins> /// Retrieve the TokenTree's span. pub fn span(&self) -> Span { match *self {

t1.next().is_none() && t2.next().is_none() } <ins> // See comments in `interpolated_to_tokenstream` for why we care about // *probably* equal here rather than actual equality // // This is otherwise the same as `eq_unspanned`, only recursing with a // different method. pub fn probably_equal_for_proc_macro(&self, other: &TokenStream) -> bool { let mut t1 = self.trees(); let mut t2 = other.trees(); for (t1, t2) in t1.by_ref().zip(t2.by_ref()) { if !t1.probably_equal_for_proc_macro(&t2) { return false; } } t1.next().is_none() && t2.next().is_none() } </ins> /// Precondition: `self` consists of a single token tree. /// Returns true if the token tree is a joint operation w.r.t. `proc_macro::TokenNode`. pub fn as_tree(self) -> (TokenTree, bool /* joint? */) {

fn test1() { let a: i32 = "foo"; //~^ ERROR: mismatched types <ins> let b: i32 = "f'oo"; //~^ ERROR: mismatched types </ins> } fn test2() {