Support time modifiers in have_been_called_with matcher