adlnet/xAPI_LRS_Test — pull request #11 by brianjmiller (2014-12-02 20:17:07+00:00)
Title: Initial framework and significant chunk of coverage

This is a pretty substantial body of work to submit as a single pull request but we wanted to get to a technically significant development point before sharing to make sure the overall direction was sound. This PR contains a significant amount of infrastructure for testing an LRS' conformance using the primary framework as previously discussed by the conformance team (with the exception of swapping Mocha for Jasmine due to better Yadda support).

There is a pretty significant README with lots of good information on installing, configuring, and running the test suite in whole and parts. Look there for more technical details.

This is by no means a finished suite, and there are parts of it that are based on interpretations of the specification language and/or intent of the ongoing discussions of the spec group. Those can (and will need to) be reviewed and adjusted to best align with the desires of the community.

Some significant items of note:

* There is not an approachable way to go back to the specification from an individual test scenario to identify what that scenario is attempting to target.
* Most of the coverage is of positive handling by the LRS, where there is negative testing there are enumerable edge cases that can be added.
* It is 1.0.x only.
* There is not a great way to deal with permissed vs. non-permissed requests and/or verification of the "authority" property.
* There is little to no coverage of "IE Mode Request" or HTTP emulation via POST requests.
* There is no coverage of HEAD requests.
* There is no OAuth coverage.
* There is little to no coverage of the read only document resources.
* Statement voiding is used, but not well covered.
* Queries of the statement stream may not be hitting good coverage of the statement targeting rules.
* There is not a great, simple summary of pass/fail of the suite as a whole. Pass/fail is indicated by all tasks running cleanly.
* It doesn't currently run in a browser. The requirement of eventual consistency with respect to the statement stream and the need for a local storage mechanism was beyond our immediate concern to deal with cross browser compatibility issues and development hassles. There is nothing specifically preventing future support of browser environments, but it would take significant work to accomplish.

Next steps:

* We need to have some discussion on the viability of keeping the work in this PR and whether it is "correct" enough that it can be merged as is, some items fixed and then merged, or whether we want to have significant discourse on it before merging. Merging and then working to fix the issues may be the easiest (best?) path since there isn't significant prior work in the repo and it will allow the targeting of specific issues via the issue tracker, and smaller pull requests to be merged. (We are happy to accept PRs to our branch to get included in this PR if it lingers.)
* Get it merged with whatever path makes the most sense from the above.
* Make sure that versioning, tagging, and releasing of the code itself is handled in a best practice software development mechanism to maintain the consistency and quality of the code. Right now the version set in the package.json tracks at 0.0.1.

Please be considerate of hosted platforms when running the test suite, it is resource intensive. A better approach to just running it may be to contact the hosted platform's support staff to ask about their current conformance and/or roadmap towards conformance.

We look forward to all the feedback and assistance from the great xAPI community!