Fix deadlock on IPFilter, some refactoring