Only remove unused imports in aggressive mode