Try to make external filters unloading less aggressive.

This commit fixes #3345 #3468.