Ensure we always set the modified time.

This is what tests get you - add a simple test, expect it to be fine, end up patching....