Remove the memoized factory when replacing the module