Only :diffoff our own diffs

Closes #395.