On second thought, don't twiddle isearch-mode-map directly.