Allow patching only annotations