Stop the card thread and make sure it's removed from the thread map when cards expire