There seems to be an error path problem here.  The called function, __wt_txn_modify, can return an error if the __wt_realloc_def call fails (in which case txn->mod_count has not been changed) or if __wt_row_key_get fails (in which case txn_mod_count has been incremented and the op is partially initialized).  The caller, say, __wt_row_modify, will not call __wt_txn_unmodify in either case.  So, if __wt_row_key_get fails, we end up with the partial op in the array.  I think this needs to call __wt_txn_unmodify here if that call fails, before returning ret.