travis-ci/travis-build — pull request #26 by rkh (2012-07-31 17:49:45+00:00)
Title: When fetching a ref, use FETCH_HEAD instead of hash for checkout

Imagine a repository with a large test suite. To reduce runtime, the developers
split the test suite into two jobs using an `env` matrix like this:

``` yaml
env:
- sub=1
- sub=2
```

The repository is rather active.

A pull request is opened, PR #42.

The repository that the pull request is sent from is called the base repository.
The repository that is supposed to be pulled from is called the head repository.

What we do now is:

A) Queue a config job:
A1) check if commit is mergeable
A2) resolve refs/pull/42/merge_head to hash X
A3) queue job B and C

B) Config job for sub=1
B1) git clone base repo
B2) git fetch +refs/pull/42/merge_head:
B3) git checkout -qf X
B4) run tests

C) Config job for sub=2
C1) git clone base repo
C2) git fetch +refs/pull/42/merge_head:
C3) git checkout -qf X
C4) run tests

Now, for an active repository, the Pull Request can be updated during this
process, which currently leads to unresolvable commits.

Any of these properties might change:

* mergeable
* sub=1 passing
* sub=2 passing

This can be provoked by either a head update or a base update.
The question is: Is the Travis behavior acceptable if this happens.

It gives us the following state changes:

```
1 passing, 2 passing => 1 passing, 2 passing
1 passing, 2 passing => 1 passing, 2 failing
1 passing, 2 passing => 1 failing, 2 passing
1 passing, 2 passing => 1 failing, 2 failing
1 passing, 2 failing => 1 passing, 2 passing
1 passing, 2 failing => 1 passing, 2 failing
1 passing, 2 failing => 1 failing, 2 passing
1 passing, 2 failing => 1 failing, 2 failing
1 failing, 2 passing => 1 passing, 2 passing
1 failing, 2 passing => 1 passing, 2 failing
1 failing, 2 passing => 1 failing, 2 passing
1 failing, 2 passing => 1 failing, 2 failing
1 failing, 2 failing => 1 passing, 2 passing
1 failing, 2 failing => 1 passing, 2 failing
1 failing, 2 failing => 1 failing, 2 passing
1 failing, 2 failing => 1 failing, 2 failing
```

There are no state changes from "unmergeable", as we don't queue jobs in that
case. State changes to "unmergeable" are irrelevant, as the maintainer will have
to test these manually anyways.

There are three strategies we want to consider:

1. Status Quo: We ignore that the PR might be updated.
2. FETCH_HEAD: This commit (a one line change)
3. Correct merge commits: We use some means to make sure we get the merge
commit exactly like it was at the point the event was triggered.

Different ways to implement the last strategy are imaginable:

* Github could implement it for us (which they already told us they wont do)
* We could merge ourself (complex, might result in issues with permissions)
* We could use the Github API to retrieve a ton of patch files (is this even
possible? and again, permission issues)

All other scenarios can be mapped to this easily.

**Any state changes only matter on base updates, as head updates will trigger a
new test run anyways.**

Since we only claim to have tested the old merge commit, reporting the old test
result is **acceptable**. However, has this is likely to influence the merge
decision, reporting the new state is **ideal**.

1. passing - **ideal**
2. passing - **ideal**
3. passing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. passing - **ideal**
2. passing - **ideal**
3. passing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. passing - **ideal**
2. passing - **ideal**
3. passing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. passing - **ideal**
2. passing - **ideal**
3. passing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. passing - **ideal**
3. passing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. passing - **acceptable**

1. error - **unacceptable**
2. failing - **ideal**
3. passing - **acceptable**

1. error - **unacceptable**
2. failing - **ideal**
3. passing - **acceptable**

1. error - **unacceptable**
2. passing - **ideal**
3. failing - **acceptable**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. passing - **ideal**
3. failing - **acceptable**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. passing - **ideal**
3. failing - **acceptable**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. passing - **ideal**
3. passing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. passing - **acceptable**

1. error - **unacceptable**
2. passing - **acceptable**
3. passing - **acceptable**

1. error - **unacceptable**
2. failing - **ideal**
3. passing - **acceptable**

1. error - **unacceptable**
2. passing - **ideal**
3. failing - **acceptable**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. passing - **unacceptable**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **acceptable**
3. failing - **acceptable**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **acceptable**
3. failing - **acceptable**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. error - **unacceptable**
2. failing - **ideal**
3. failing - **ideal**

1. passing - **ideal**
2. passing - **ideal**
3. passing - **ideal**

1. passing - **acceptable**
2. passing - **acceptable**
3. passing - **acceptable**

1. passing - **acceptable**
2. passing - **acceptable**
3. passing - **acceptable**

1. passing - **acceptable**
2. passing - **acceptable**
3. passing - **acceptable**

1. failing - **acceptable**
2. failing - **acceptable**
3. failing - **acceptable**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **acceptable**
2. failing - **acceptable**
3. failing - **acceptable**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **acceptable**
2. failing - **acceptable**
3. failing - **acceptable**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

1. failing - **ideal**
2. failing - **ideal**
3. failing - **ideal**

Because a new commit to head will happen to fix the issue, which will trigger
a new build.

* ideal: 26
* acceptable: 6
* unacceptable: 32

* ideal: 54
* acceptable: 9
* unacceptable: 1

* ideal: 46
* acceptable: 18
* unacceptable: 0

The FETCH_HEAD strategy results in the most ideal outcomes. While there is one
unacceptable outcome, it is extremely unlikely: A commit that fixes an issue
in the Pull Request (but not in base) with sub=2 and at the same time causes an
issue in that Pull Request (but not in base) in sub=1 has to happen after we
did the clone for sub=1 but before we did the clone for sub=2. In this rare case
the issue will be detected after pressing the merge button.