No. load_dir is called whenever a cache is initialized.  So the default cache is loaded when the spec_helper is loaded, then each named cache will be loaded when it is first used.