Fix use-after-free when patch outlives diff

And added test that would trigger the bug in Valgrind.
Aren't we leaking the patch now?