Revert "Avoid recreating all metrics on reload"

This reverts commit ab8d36b3973364b3a87f90767b9fadab016ac38b.