edx/edx-platform — pull request #678 by JonahStanley (2013-08-14 20:06:22+00:00)
Title: Zoldak/acceptance sauce

Suggested Reviewers: @wedaly @jzoldak

If everything looks good, let me know and I will try to configure my own jenkins to actually test multi-configuration to ensure it works before merging in. I want to get a preliminary look at it to hopefully avoid the time sink of going through multiple jenkins iterations. If nothing is changed on jenkins or the environment variables are not set, the default behavior remains.

This is everything that is needed to allow our acceptance tests to connect to sauce labs and to run the tests in a matrix of browsers. In order to do this, the following was added:

1. browser.py will now check the settings to see if it is a sauce connection and will properly set up the desired capabilities. If sauce is not enabled, the tests will proceed as normal. Also, the functionality to update the pass/fail status on sauce is included.

2. The settings are obtained through environment variables that are sent hopefully through jenkins. It is also important to note that sauce can only be connected to on a small list of ports. In keeping with the idea of having a random port (so acceptance and unit tests do not clash), the acceptance tests will choose a random port from this list.

In order to utilize sauce properly, the following variables must be set by jenkins
ENABLED : true if using sauce
SAUCE_USER_NAME, SAUCE_API_KEY : credentials for accesing sauce
SAUCE_PLATFORM, SAUCE_BROWSER, SAUCE_VERSION: specifications for what sauce is running
SAUCE_DEVICE only applies to one set of sauce configurations and doesn't need to be configured
The build name will correspond to the git commit hash found through JOB_NAME

3. Some tests are known to not work in Sauce or on specific browsers, those scenarios are annotated with tags so they can be skipped.

4. test_acceptance.sh was changed so that if sauce or a browser type is specified, it will skip all of the tests that do not work on sauce or that browser type.

Lastly, we should be able to specify what browser we want jenkins to run acceptance tests under. Acceptance.py will now take either the environment variable 'LETTUCE_BROWSER' or use a default of 'chrome'. This is so there can be jobs for multiple browsers that do not need to run through sauce and can be configured easily. test_acceptance.sh also reflects this by also allowing the possibility to skip tests based on this variable