Limit the scope of `with-silent-modifications'

Fixes #188