Core: Change hole filling to perform filtering on demand

This should ideally use less memory.