Core: Remove the warmer listener when the FixedBitSetFilterCache gets closed.