refs.c: add transaction support for replacing a reflog

Add a new transaction function transaction_replace_reflog.
This function takes a blob and replaces the new or existing reflog with the
content of this blob. This will be used by rename_ref where we basically want
to copy the existing blob as is.

Change-Id: I7a38cfbfdd9927adde6a458880750d370eabcd2a
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: add new function copy_reflog_into_strbuf

Add a new function copy_reflog_into_strbuf() that is a backend specific
optimized way to read the whole reflog as is into a strbuf so we can
copy it elsewhere instead of having to iterate over all the entries using
the iterators.

For the current type of refs backend that use files we simple read_in_full
the whole file into a strbuf. When we add other types of backends for refs
we will need to implement similarly optimized versions of this functions for
them.

Change-Id: I6f718076082f35fbadcf1d3b586ae038a55db669
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: update rename_ref to use a transaction

Change refs.c to use a single transaction to perform the rename.
Change the function to return 1 on failure instead of either -1 or 1.

These changes make the rename_ref operation atomic.

Change-Id: I7ef74c3861bfbccc4d5a7f3b8ca210118aacc62a
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: rollback the lockfile before we die() in repack_without_refs

commit 3989c2a763c3b355785d609b3144c7935dffb273 upstream.

Change-Id: I63a22da521ecc8eb60d7a8aaa5af666d2827a599
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
refs.c: move reflog updates into its own function

write_ref_sha1 tries to update the reflog while updating the ref.
Move these reflog changes out into its own function so that we can do the
same thing if we write a sha1 ref differently, for example by writing a ref
to the packed refs file instead.

No functional changes intended. We only move some code out into a separate
function.

Change-Id: I4aeac7812472035797497c405127f39561d4603e
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: write updates to packed refs when a transaction has more than one ref

When we are updating more than one single ref, i.e. not a commit, then
write the updated refs directly to the packed refs file instead of writing
them as loose refs.

Change clone to use a transaction instead of using the packed refs API.
This changes the behavior of clone slightly. Previously clone would always
clone all refs into a packed refs file. With this change clone will only
clone into packed refs iff there are two or more refs being cloned.
If the repository we are cloning from only contains exactly one single ref
then clone will now store this as a loose ref. The benefit here is that
we no longer need to export a bunch of API functions to clone to access
packed refs directly. Clone can now just use a normal transaction and all
the packed refs goodness will happen automatically.

Update the t5516 test to cope with the fact that clone now only uses
packed refs if there are more than one ref being cloned.

We still use loose refs for single ref transactions, such as are used
by 'git commit' and friends. The reason for this is that if you have very
many refs then having to re-write the whole packed refs file for every
common operation like commit would have a performance impact.
That said, with these changes it should now be fairly straightforward to
add support to optionally start using packed refs for ALL updates
which could solve existing issues with name clashes in case insensitive
filesystems.

This change also means that multi-ref updates will now appear as a single
atomic change to any external observers instead of a sequence of discreete
changes.

Change-Id: I9894bc8ae1b4dd69ed6c51e5b1d48202845d5fc4
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
remote.c: use a transaction for deleting refs

Transactions now use packed refs when deleting multiple refs so there is no
need to do it manually from remote.c any more.

Change-Id: I58f6f4e569a7533fcc8704b8bc9aeabeed19d93e
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: make repack_without_refs static

Change-Id: Iaa775ea000934c72989ed1acb3948a7acee12a55
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: make the *_packed_refs functions static

We no longer need to expose the lock/add/commit/rollback functions
for packed refs anymore so make them static and remove them from the
public api.

Change-Id: I8fe63f89a0aef6a3ebef38fab3f528b876ede890
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: replace the onerr argument in update_ref with a strbuf err

Get rid of the action_on_err enum and replace the action argument to
update_ref with a strbuf *err for error reporting.

Update all callers to the new api including two callers in transport*.c
which used the literal 0 instead of an enum.

Change-Id: I9d49c9cfae5988f8d333971bc77c8183f8509a36
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: make add_packed_ref return an error instead of calling die

Change add_packed_ref to return an error instead of calling die().
Update all callers to check the return value of add_packed_ref.

Change-Id: I2c018959067acb78c50212d903287b381efd8c25
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: make lock_packed_refs take an err argument

Change-Id: I57fda2e3431194c99e066feee86ae49579c579ba
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: add an err argument to pack_refs

Change-Id: I36f4c08c2cf98f94c1455e894e7f121b63abecde
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
receive-pack.c: add protocol support to negotiate atomic-push

This adds support to the protocol between send-pack and receive-pack to
* allow receive-pack to inform the client that it has atomic push capability
* allow send-pack to request atomic push back.

There is currently no setting in send-pack to actually request that atomic
pushes are to be used yet. This only adds protocol capability not ability
for the user to activate it.

Change-Id: I37530b96c8f5288130ee5e05395a8cda03ba470b
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
send-pack.c: add an --atomic-push command line argument

This adds support to send-pack to to negotiate and use atomic pushes
iff the server supports it. Atomic pushes are activated by a new command
line flag --atomic-push.

In order to do this we also need to change the semantics for send_pack()
slightly. The existing send_pack() function actually don't sent all the
refs back to the server when multiple refs are involved, for example
when using --all. Several of the failure modes for pushes can already be
detected locally in the send_pack client based on the information from the
initial server side list of all the refs as generated by receive-pack.
Any such refs that we thus know would fail to push are thus pruned from
the list of refs we send to the server to update.

For atomic pushes, we have to deal thus with both failures that are detected
locally as well as failures that are reported back from the server. In order
to do so we treat all local failures as push failures too.

We introduce a new status code REF_STATUS_ATOMIC_PUSH_FAILED so we can
flag all refs that we would normally have tried to push to the server
but we did not due to local failures. This is to improve the error message
back to the end user to flag that "these refs failed to update since the
atomic push operation failed."

Change-Id: I1444d896f6bc6b6f1078b893b5a5b5186b445159
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
receive-pack.c: use a single transaction when atomic-push is negotiated

Update receive-pack to use an atomic transaction iff the client negotiated
that it wanted atomic-push.
This leaves the default behavior to be the old non-atomic one ref at a
time update. This is to cause as little disruption as possible to existing
clients. It is unknown if there are client scripts that depend on the old
non-atomic behavior so we make it opt-in for now.

Later patch in this series also adds a configuration variable where you can
override the atomic push behavior on the receiving repo and force it
to use atomic updates always.

If it turns out over time that there are no client scripts that depend on the
old behavior we can change git to default to use atomic pushes and instead
offer an opt-out argument for people that do not want atomic pushes.

Change-Id: I233594fbff357383b794230672beab7933e3ce0f
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
push.c: add an --atomic-push argument

Add a command line argument to the git push command to request atomic
pushes.

Change-Id: I2e17cbd4d53f7f8bd55f0477363af6173522e09a
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
t5543-atomic-push.sh: add basic tests for atomic pushes

Change-Id: I16bb0b85a7f780e1e5b55b90d13784662d20e701
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
receive-pack.c: add a receive.preferatomicpush configuration variable

Add receive.preferatomicpush setting to receive-pack.c. This triggers
a new capability "prefer-atomic-push" to be sent back to the send-pack
client, requesting the client, if it supports it, to request
an atomic push.

This is an alternative way to trigger an atomic push instead of having
to rely of the user using the --atomic-push command line argument.
For backward compatibility, this is only a hint to the client that is should
request atomic pushes if it can. Old clients that do not support atomic pushes
will just ignore this capability and use a normal push.

The reason we need to signal this capability back to the client is due
to that send_pack() has push failure modes where it will detect that
certain refs can not be pushed and fail them early. Those refs would not
even be sent by the client unless atomic-pushes are activated.
This means that IF we activate this feature from the server side we must
tell the client to not fail refs early and use an atomic push. We can not
enforce this on the server side only.

Change-Id: Ic3b0ee526f246b78c21d49d861af3a4d38dba1e5
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
refs.c: add an err argument to create_reflog

Add err argument to create_reflog that can explain the reason for a
failure. This then eliminates the need to manage errno through this
function since we can just add strerror(errno) to the err string when
meaningful. No callers relied on errno from this function for anything
else than the error message.

log_ref_write is a private function that calls create_reflog. Update
this function to also take an err argument and pass it back to the caller.
This again eliminates the need to manage errno in this function.

Update the private function write_sha1_update_reflog to also take an
err argument.

Change-Id: I0a69959b47f9261e77beac1d1923ad408eee48f9
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>