Change:
In investigating I was generating some source code, and using the shell script listed in we can see: That's a lot of time to expand a crate that doesn't have any macros in it!
cc you may be interested in this!
A looks like almost all the time is spent in In which case, cc
Apparently most time is spent
called ~38k times
Since we need to build the module graph and resolve imports during expansion for macros 2.0, we must construct node ids and def ids throughout expansion (i.e. interleaved arbitrarily many times until we hit fix-point). Ideally we'd keep running totals of the time spent assigning node ids, assigning def ids, etc. for more accurate profiling information within the expansion step.
in theory though isn't the fixed point for this crate the crate itself? (e.g. there's no macros here). Or are the def ids persisted for the rest of compilation, so "blaming" expansion isn't quite correct here?
Yeah, the def ids are persisted for the rest of compilation. We could move the "first round" of def id collection (in which we collect the raw, unexpanded source) outside of expansion, but that would introduce more code complexity and wouldn't count the time we spend collecting def ids for AST that comes from macro expansions.
Ah ok, nah it seems fine! I think there's a number of cases where is attributing time in the wrong place nowadays... Still it seems odd that 30-50k invocations of this function would take so long to execute?
Do I understand this correctly that all DefIds generated are actually used, right? There are no "temporary" DefIds? Each of these invocations creates a Blake2 hasher. I'd be interested how the performance looks with other hashing algorithms. I'd like to switch to something faster anyway:
Are you sure this isn't the other quadratic case fixed in ?
It may very well be! Feel free to close this w/ that PR landing, I'll verify to be sure but I'd imagine you're correct in that it's fixed by
er, didn't mean to close just yet

Discussion:
use hir::def_id::{CrateNum, DefId, DefIndex, LOCAL_CRATE, DefIndexAddressSpace, CRATE_DEF_INDEX}; use ich::Fingerprint; <del> use rustc_data_structures::fx::{FxHashMap, FxHashSet}; </del> <ins> use rustc_data_structures::fx::FxHashMap; </ins> use rustc_data_structures::indexed_vec::IndexVec; use rustc_data_structures::stable_hasher::StableHasher; use serialize::{Encodable, Decodable, Encoder, Decoder};

pub(super) node_to_hir_id: IndexVec<ast::NodeId, hir::HirId>, macro_def_scopes: FxHashMap<Mark, DefId>, expansions: FxHashMap<DefIndex, Mark>, <del> keys_created: FxHashSet<DefKey>, </del> <ins> next_disambiguator: FxHashMap<(DefIndex, DefPathData), u32>, </ins> } // Unfortunately we have to provide a manual impl of Clone because of the

node_to_hir_id: self.node_to_hir_id.clone(), macro_def_scopes: self.macro_def_scopes.clone(), expansions: self.expansions.clone(), <del> keys_created: self.keys_created.clone(), </del> <ins> next_disambiguator: self.next_disambiguator.clone(), </ins> } } }

node_to_hir_id: IndexVec::new(), macro_def_scopes: FxHashMap(), expansions: FxHashMap(), <del> keys_created: FxHashSet(), </del> <ins> next_disambiguator: FxHashMap(), </ins> } }

// The root node must be created with create_root_def() assert!(data != DefPathData::CrateRoot); <del> // Find a unique DefKey. This basically means incrementing the disambiguator // until we get no match. let mut key = DefKey { </del> <ins> // Find the next free disambiguator for this key. let disambiguator = { let next_disamb = self.next_disambiguator.entry((parent, data.clone())).or_insert(0); let disambiguator = *next_disamb; *next_disamb = next_disamb.checked_add(1).expect("disambiguator overflow"); disambiguator }; let key = DefKey { </ins> parent: Some(parent), disambiguated_data: DisambiguatedDefPathData { <del> data, disambiguator: 0 </del> <ins> data, disambiguator </ins> } }; <del> while self.keys_created.contains(&key) { key.disambiguated_data.disambiguator += 1; } self.keys_created.insert(key.clone()); </del> let parent_hash = self.table.def_path_hash(parent); let def_path_hash = key.compute_stable_hash(parent_hash);

sp } fn ensure_filemap_source_present(&self, file_map: Rc<FileMap>) -> bool { <del> let src = self.file_loader.read_file(Path::new(&file_map.name)).ok(); return file_map.add_external_src(src) </del> <ins> file_map.add_external_src( || self.file_loader.read_file(Path::new(&file_map.name)).ok() ) </ins> } }

/// If the hash of the input doesn't match or no input is supplied via None, /// it is interpreted as an error and the corresponding enum variant is set. /// The return value signifies whether some kind of source is present. <del> pub fn add_external_src(&self, src: Option<String>) -> bool { </del> <ins> pub fn add_external_src<F>(&self, get_src: F) -> bool where F: FnOnce() -> Option<String> { </ins> if *self.external_src.borrow() == ExternalSource::AbsentOk { <ins> let src = get_src(); </ins> let mut external_src = self.external_src.borrow_mut(); if let Some(src) = src { let mut hasher: StableHasher<u128> = StableHasher::new();