[SPARK-4048] Enhance and extend hadoop-provided profile.

This change does a few things to make the hadoop-provided more useful:

- Create new profiles for other libraries / services that might be provided by the infrastructure
- Simplify and fix the poms so that the profiles are only activated while building assemblies.
- Fix tests so that they're able to run when the profiles are activated
- Add a new env variable to be used by distributions that use these profiles to provide the runtime
  classpath for Spark jobs and daemons.
So, streaming does not depend on Jetty, but it does use servlet APIs. Since Spark doesn't pull in any servlet API dependency directly, and instead uses the transitive one provided by Jetty, which is some Jetty repackaging of servlet APIs instead of some common dependency. (BTW it works without the explicit dependency because now it was coming as a transitive dependency of spark-core.)

In light of that I'll restore this dependency, even though I don't really like it very much (it's pulling a dependency that is too coarse for what it needs).