rackerlabs/developer.rackspace.com — pull request #67 by smashwilson (2014-04-28 19:44:30+00:00)
Title: [wip] Toward automated code sample validation

One of the big problems with keeping a guide up to date is dealing with stale code. SDKs change and APIs change, and stale documentation can lead to a situation where someone's first encounter with the SDK is something you copy-and-paste that breaks in mysterious ways, which instantly makes you suspicious about the accuracy of the rest of the guide. Speaking for myself, that's the point when I start ignoring the docs and reading the source.

My goal here is to create a framework that will extract code samples from the documentation, plug them in to a templated file that supplies boilerplate and makes assertions, and then create an easy way to run them all and get a nice report. As a bonus, I want to make it self-check for completeness, so having a code sample in the docs that's not referenced in the template will be an error. I'm using `.erb` files for templating, so it looks something like this:

```erb
<%= inject :create_container %>

refute_nil @client.directories.get('sample-container-test')

<%= inject :set_container_as_cdn %>

assert directory.public?

# ...
```

Offhand, the Javascript samples will cause the most problems because they'll need to deal with callbacks somehow. For those, I'd like to offer the ability to inject code into the middle of a snippet at a `TODO` mark by providing a block:

```erb
<%= inject :auth_client %>

<%= inject :create_container do |todo| %>
<% case todo %>
<% when "handle as appropriate" %>
throw new Error(err);
<% when "use your container" %>
<%= inject :upload_object %>
// ... etc
// Alternately, tie in to a promise library like Q here.
<% else unrecognized(todo) %>
<% end %>
<% end %>
```

## Progress

- [x] Create a full template for a single service for a single language.
- [x] Provide credential management.
- [ ] Generalize to work for all services, code samples, and languages.
- [ ] Add a build step for Java and C# samples.
- [ ] Provide a directory for support code that's useful for all templates of a particular language.
- [ ] Find a reasonable, less ugly solution to the Javascript callback problem.
- [ ] Report output in some fancy way. :art:
- [ ] Notice unimplemented code blocks and services. Report them and fail.
- [ ] Invoke it from a `rake` task at the top-level.
- [ ] Support clean-up actions to be taken after a service is executed, pass or fail.