Don't attach filter delay unless filtering is present.

This avoids accidentally binding to random inputs on the page and breaking things. Yes, it's true that you should just not activate the filter delay unless you've got filtering enabled, but in the interest of the best possible default experience, that shouldn't cause side effects.