I took a quick look in the codex, if the filter isn't set the remove will be a no-op.  DIfference is that the return value from the remove will be false instead of true.