Refactor selector matching

No behavior change (hopefully).

a16087d is the core of what I wanted to do, following up on #1464 (cc: @parkjaeman)