Properly age out client server cache items.

Was a logic bug and iterator + map removal combo
that resulted in non-deletion.