Model selection and refactor to keep experts, relevant experts in memory