Change:
The script below has a macro_rules macro that expands to: But inside of the custom derive the tokens received are: The seems to be missing. Originally reported in rustc 1.28.0-nightly ( 2018-06-02) Repro script:
The behavior is correct with rustc 1.15.0 but the current incorrect behavior exists since 1.16.0.
cc — perhaps somebody could bisect this to a specific nightly? (I guess it's too old to get a specific PR) may help here :)
Also, how does it come to happen that is still in there and not replaced during macro expansion...? Maybe I'm unclear on the phasing of how works.
Marking as P-high but don't have an assignee yet, will revisit next week when we've hopefully had a chance to bisect.
2016-12-06 has the odd result of . 2016-12-27 is the first nightly to have as the result; the previous nightly 2016-12-23 has the (correct, I presume) result.
These are the bors commits between those two nightlies (). Auto merge of - elahn:windows-readconsole-ctrl-z, r=alexcrichton Auto merge of - japaric:do-not-delete-enable-llvm-backend, r=alexcrichton Auto merge of - YaLTeR:fastcall-fix, r=pnkfelix Auto merge of - petrochenkov:altname, r=jseyfried Auto merge of - retep998:flauschige-kaninchen, r=petrochenkov Auto merge of - brson:em, r=alexcrichton Auto merge of - jseyfried:defidvis, r=nrc Auto merge of - jseyfried:fixresolvehang, r=eddyb Auto merge of - jseyfried:fiximportresolutionbug, r=eddyb Auto merge of - steveklabnik:rollup, r=steveklabnik Auto merge of - frewsxcv:file-docs, r=brson Auto merge of - alexcrichton:fix-line-writer, r=brson Auto merge of - withoutboats:parsewherehigherrankhack, r=eddyb Auto merge of - camlorn:disablefieldreordering, r=nikomatsakis Auto merge of - jseyfried:legacycustomderivedeprecation, r=nrc Auto merge of - nrc:save-sig, r=nikomatsakis Auto merge of - Mark-Simulacrum:drop-glue, r=eddyb Auto merge of - redox-os:redoxcross, r=brson Auto merge of - jseyfried:refactorglobal_paths, r=nrc Auto merge of - brson:rm-llvm-lock, r=brson I think , , , , are possible candidates based on branch names. I'll try to run git bisect in this commit range.
I was.. largely unsuccessful at running git bisect. Maybe someone else will have more luck.
I figure it's easier to bisect the Bors commits which are sure to build correctly, rather than to run . Overnight I ran a build of , and got the erroneous output, cutting it down to these: Auto merge of - jseyfried:legacycustomderivedeprecation, r=nrc Auto merge of - nrc:save-sig, r=nikomatsakis Auto merge of - Mark-Simulacrum:drop-glue, r=eddyb Auto merge of - redox-os:redoxcross, r=brson Auto merge of - jseyfried:refactorglobalpaths, r=nrc Auto merge of - brson:rm-llvm-lock, r=brson My money's on but I won't be able to run another build until I get to work. <details<summarySlightly modified repro script for working with such old versions of rust</summary</details
Okay, I have confirmed that exhibits the issue, and does not, so it is . cc
The following test case (replacing the similarly-named macro in the OP) helps clarify the nature of the bug. It is not "overwriting" the path component, but rather, it is missing the entire last path component (with type parameters): The is dropped: (Note: tested on nightly, because I foolishly did not keep my build of ) Edit: Additionally, only the trait path is affected. Adding before works fine. The PR is largely one refactoring commit. I've been reading through looking for the kinds of things that might enable a bug like this, but haven't found the culprit.
I have a theory. I believe this was buggy prior to jseyfried's PR, but the PR caused the bug to affect this specific test case. The following function (used exclusively by ) uses a to normalize away from paths: However, the trait path is stored in an . Prior to the PR, a path segment was deleted if (and only if) the macro was local. After the PR, a path segment is inserted if (and only if) the macro was from another crate. In each of these cases, the QSelf position field is invalidated. If my theory is correct, then prior to , it should have manifested as printing extra tokens if the definition of is moved from to . (However I no longer have the build, and I'm not sure where is successfully downloading ancient nightlies from!)
I'm downloading nightlies via cargo-bisect-rustc's bisection; I also found that for some reason rustup itself refuses to install those nightlies.
Thanks! I was able to verify my prediction; handles the original test case correctly, but duplicates the when the macro is defined locally: <details<summaryRepro with local and nonlocal macros</summary</detailsI'll start working on a PR.
Silly question: Would adding an AST type like be overkill? This pattern describes all appearances of in the AST, and I feel the most principled solution to this bug would be to add a method for this pairing of types. (whereas if I just overrode , , and in the one function linked above, it could easily get neglected the next time QSelf is somewhere) Edit: It appears that such a type in fact used to exist (), which kind of explains the bizarre name of . It was removed in for some reason. Edit 2: also perhaps libsyntax is not as malleable as I thought, due to a number of rustc plugin crates that still exist...
Since I wasn't sure about the above, I wrote two possible fixes, both of which I am now testing: A self-contained fix that makes override , but it's messy A cleaner fix that involves adding the following method to :
New delegating to by default (without changing qpaths in AST) looks like a good solution.
Thanks for the feedback! .

Discussion:
struct EliminateCrateVar<'b, 'a: 'b>(&'b mut Resolver<'a>, Span); impl<'a, 'b> Folder for EliminateCrateVar<'a, 'b> { <del> fn fold_path(&mut self, mut path: ast::Path) -> ast::Path { </del> <ins> fn fold_path(&mut self, path: ast::Path) -> ast::Path { match self.fold_qpath(None, path) { (None, path) => path, _ => unreachable!(), } } fn fold_qpath(&mut self, mut qself: Option<ast::QSelf>, mut path: ast::Path) -> (Option<ast::QSelf>, ast::Path) { qself = qself.map(|ast::QSelf { ty, path_span, position }| { ast::QSelf { ty: self.fold_ty(ty), path_span: self.new_span(path_span), position, } }); </ins> let ident = path.segments[0].ident; if ident.name == keywords::DollarCrate.name() { path.segments[0].ident.name = keywords::CrateRoot.name();

ast::Ident::with_empty_ctxt(name).with_span_pos(span) ), _ => unreachable!(), <del> }) </del> <ins> }); if let Some(qself) = &mut qself { qself.position += 1; } </ins> } } <del> path </del> <ins> (qself, path) </ins> } fn fold_mac(&mut self, mac: ast::Mac) -> ast::Mac {

noop_fold_path(p, self) } <ins> fn fold_qpath(&mut self, qs: Option<QSelf>, p: Path) -> (Option<QSelf>, Path) { noop_fold_qpath(qs, p, self) } </ins> fn fold_path_parameters(&mut self, p: PathParameters) -> PathParameters { noop_fold_path_parameters(p, self) }

TyKind::Tup(tys) => TyKind::Tup(tys.move_map(|ty| fld.fold_ty(ty))), TyKind::Paren(ty) => TyKind::Paren(fld.fold_ty(ty)), TyKind::Path(qself, path) => { <del> let qself = qself.map(|QSelf { ty, path_span, position }| { QSelf { ty: fld.fold_ty(ty), path_span: fld.new_span(path_span), position, } }); TyKind::Path(qself, fld.fold_path(path)) </del> <ins> let (qself, path) = fld.fold_qpath(qself, path); TyKind::Path(qself, path) </ins> } TyKind::Array(ty, length) => { TyKind::Array(fld.fold_ty(ty), fld.fold_anon_const(length))

} } <ins> pub fn noop_fold_qpath<T: Folder>(qself: Option<QSelf>, path: Path, fld: &mut T) -> (Option<QSelf>, Path) { let qself = qself.map(|QSelf { ty, path_span, position }| { QSelf { ty: fld.fold_ty(ty), path_span: fld.new_span(path_span), position, } }); (qself, fld.fold_path(path)) } </ins> pub fn noop_fold_path_parameters<T: Folder>(path_parameters: PathParameters, fld: &mut T) -> PathParameters {

PatKind::TupleStruct(folder.fold_path(pth), pats.move_map(|x| folder.fold_pat(x)), ddpos) } <del> PatKind::Path(opt_qself, pth) => { let opt_qself = opt_qself.map(|qself| { QSelf { ty: folder.fold_ty(qself.ty), path_span: folder.new_span(qself.path_span), position: qself.position, } }); PatKind::Path(opt_qself, folder.fold_path(pth)) </del> <ins> PatKind::Path(qself, pth) => { let (qself, pth) = folder.fold_qpath(qself, pth); PatKind::Path(qself, pth) </ins> } PatKind::Struct(pth, fields, etc) => { let pth = folder.fold_path(pth);

lim) } ExprKind::Path(qself, path) => { <del> let qself = qself.map(|QSelf { ty, path_span, position }| { QSelf { ty: folder.fold_ty(ty), path_span: folder.new_span(path_span), position, } }); ExprKind::Path(qself, folder.fold_path(path)) </del> <ins> let (qself, path) = folder.fold_qpath(qself, path); ExprKind::Path(qself, path) </ins> } ExprKind::Break(opt_label, opt_expr) => { ExprKind::Break(opt_label.map(|label| folder.fold_label(label)),

use proc_macro::TokenStream; <ins> // Outputs another copy of the struct.  Useful for testing the tokens // seen by the proc_macro. </ins> #[proc_macro_derive(Double)] pub fn derive(input: TokenStream) -> TokenStream { format!("mod foo {{ {} }}", input.to_string()).parse().unwrap()

<ins> // Copyright 2016 The Rust Project Developers. See the COPYRIGHT // file at the top-level directory of this distribution and at // http://rust-lang.org/COPYRIGHT. // // Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or // http://www.apache.org/licenses/LICENSE-2.0> or the MIT license // <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your // option. This file may not be copied, modified, or distributed // except according to those terms. pub struct ExternFoo; pub trait ExternTrait { const CONST: u32; type Assoc; } impl ExternTrait for ExternFoo { const CONST: u32 = 0; type Assoc = ExternFoo; } #[macro_export] macro_rules! external { () => { mod bar { #[derive(Double)] struct Bar($crate::ExternFoo); } mod qself { #[derive(Double)] struct QSelf(<$crate::ExternFoo as $crate::ExternTrait>::Assoc); } mod qself_recurse { #[derive(Double)] struct QSelfRecurse(< <$crate::ExternFoo as $crate::ExternTrait>::Assoc as $crate::ExternTrait>::Assoc ); } mod qself_in_const { #[derive(Double)] #[repr(u32)] enum QSelfInConst { Variant = <$crate::ExternFoo as $crate::ExternTrait>::CONST, } } } } </ins>

// except according to those terms. // aux-build:double.rs <ins> // aux-build:external-crate-var.rs </ins> // ignore-stage1 #![allow(unused)] #[macro_use] extern crate double; <ins> #[macro_use] extern crate external_crate_var; </ins> struct Foo; <del> macro_rules! m { () => { #[derive(Double)] struct Bar($crate::Foo); </del> <ins> trait Trait { const CONST: u32; type Assoc; } impl Trait for Foo { const CONST: u32 = 0; type Assoc = Foo; } macro_rules! local { () => { // derive_Double outputs secondary copies of each definition // to test what the proc_macro sees. mod bar { #[derive(Double)] struct Bar($crate::Foo); } mod qself { #[derive(Double)] struct QSelf(<::Foo as $crate::Trait>::Assoc); } mod qself_recurse { #[derive(Double)] struct QSelfRecurse(<<$crate::Foo as $crate::Trait>::Assoc as $crate::Trait>::Assoc); } mod qself_in_const { #[derive(Double)] #[repr(u32)] enum QSelfInConst { Variant = <::Foo as $crate::Trait>::CONST, } } </ins> } } <del> m!(); </del> <ins> mod local { local!(); } // and now repeat the above tests, using a macro defined in another crate mod external { external!{} } </ins> fn main() {}