Revert "Disable removal caching in cached_level_map for now"

This reverts commit ad7aa73c9df06d3e06cf980fec77cf226d964c45.