Added logic for chunked filtering (to avoid disk space issues)