fabioz/Pydev — pull request #78 by jonahkichwacoders (2013-09-25 22:41:28+00:00)
Title: Travis CI Integration

Here is a pull request to enable PyDev to be built with Travis CI https://travis-ci.org.

I have completed the work of getting PyDev to build, test and deploy using Travis. The build and test is entirely done with Maven/Tycho so is quite portable. It worked well on my Windows machine and the main development was done on Ubuntu 12.04 64-bit (which is the same as the Travis VMs run).

High level features are:
1) Tests are run with Tycho/Maven. This includes Workbench tests and all dependencies in TestDependent met (including Jython and IronPython)
2) Maven signs jars so that the release version can be done from Travis CI
3) Travis scripts exist to setup and deploy. Deployment is implemented to deploy to an S3 bucket.
4) 2 jobs are run on Travis, one that creates deployable PyDev and one that creates testable version. The testable version is made so by applying test_with_tycho.patch.

There are some outstanding issues, listed in travis.TODO.

You can see the result of this build here (there should be a build passed confirmation at the bottom of this text too with the same link):
https://travis-ci.org/jonahkichwacoders/Pydev/builds/11800256

if you look at the bottom of the testing (PYDEV_TEST=true) log:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/11800257/log.txt
you can see links to the upload artifacts (search for Public URL of the object is:), eg.:
http://pydevbuilds2.s3.amazonaws.com/artifacts/build/11800256/11800257/junit-noframes.html
which shows 2390 Tests with 1 Error and 42 Failures

if you look at the bottom of the release (PYDEV_TEST=false) log:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/11800258/log.txt
you can see links to the upload artifacts (search for Public URL of the object is:), eg.:
http://pydevbuilds2.s3.amazonaws.com/artifacts/branch/travis_ci_matrix/release/runnable.zip
or
http://pydevbuilds2.s3.amazonaws.com/artifacts/build/11800256/11800258/runnable.zip

The artifact urls that start with branch/ are replaced with the most recent result on that branch. The branch is followed by release or test depending on which type of build was run.
The build/ artifacts are never deleted.
To find out what built, look at the build.info in the build/123/456/ or branch/abc/{test,release} directories.

For example, you can point your update site at:
http://pydevbuilds2.s3.amazonaws.com/artifacts/branch/travis_ci_matrix/release/repository/
and then it will stay up to date with the most recent commit on that branch.
(This won't work today because the build numbers are not qualified.)

To fully integrate travis, the secure environment variables need to be set to work with PyDev:
1) Log-in/setup Travis CI and enable it for Pydev repo (follow instructions on Travis)
2) Create an S3 bucket in us-east-1 (physically located next to Travis's build machines, makes build times faster)
3) Follow http://about.travis-ci.org/docs/user/encryption-keys/ to create environment variables and update the relevant ones in .travis.yml
NOTE: You don't need to do any of these steps. The things that S3 is used for:
1) Storing your keystore so you can sign jars. This is independently optional, if you don't provide a keystore in the expected place (s3://$ARTIFACTS_S3_BUCKET/pydev.keystore) then the travis build will not sign the jars.
2) Caching .m2/repository value in s3://$ARTIFACTS_S3_BUCKET/m2repo.tar.gz. This does not have to be pre-loaded, the builds will create/update it on each run. (Generally the 120MB repo downloads in 3-4 seconds this way. If you run without S3 it takes 5-30 minutes to download everything from p2 mirrors.)
3) Storing JUnit and Build results. See travisupload.sh

S3 Costs: (estimates)
- each build run costs just over $0.02 in data transfer (to download cached m2 repo at about 120 MB x 2) + $0.001 for uploading results (200 requests at most, we currently upload 30 files to S3 per build x 2 times per job x 2 jobs + we do some list requests to know what to delete)
- each build stored costs less that $0.01 per month in storage
- each branch stored costs less that $0.01 per month in storage
- 3rd party pull requests do not have any costs as they happen without secure travis variables and therefore with no access to S3

The final thing todo is update version numbers. I haven't included it in this pull request because it is unrelated. It is quite easy now to get MANIFEST.MF, feature.xml and pom.xml updated (see root pom.xml for more info). Please let me know if you would like me to submit a new pull request.
mvn tycho-versions:set-version -DnewVersion=3.0.0-SNAPSHOT

Please be in touch if there is anything I can change. Once this gets approved I will work on the remaining TODOs in travis.TODO, starting from the top.

Thanks
Jonah