Fixes referrer being matched when passed an empty array

The referrer array is queried even when it's empty. Other filters ignore
empty arrays, so this fix is more for consistency.