vignette: clip inner, unvignetted part of image (fixes #10041)

without this fix the inner area of the image preserves potentially unbounded values
while the vignette area is clipped. unter certain circumstances this leads to
visible artifacts at the transition (see redmine ticket). legacy history stacks
are handled as before. to take advantage of this fix you need to reset the
vignette module.