Refactor some search stuff

We're not going to send it through if it's not in `filters` so it just gets ignored anyway. I felt like that didn't add much but did add to the code complexity.