On which backend have you tested this? I don't see how this can work on sequential and openmp backends, I'm probably missing something...
Anyway, could you rather use the IR machinery? Otherwise, if I understood it correctly, this is going to have the "usual" performance issues on CPUs.