Queue up lazy insertions to graph correctly (might be slow)