inpaint: Remove unneeded mutex and extra loop

Mutex was for back when we still had a bug in our caching system. The
extra loop removed was the per element loop. We can do everything in a
single pass.