Only make the factory if we need to