Fix logic for filtered scored queries

* The logic as is currently 'works'
* The logic is really messy and in need of refactor
* My current attempts to refactor it have failed miserably