Clearing callback latch map after cancelling the callbacks.