[SPARK-1768] History server enhancements.

Two improvements to the history server:

- Separate the HTTP handling from history fetching, so that it's easy to add
  new backends later (thinking about SPARK-1537 in the long run)

- Avoid loading all UIs in memory. Do lazy loading instead, keeping a few in
  memory for faster access. This allows the app limit to go away, since holding
  just the listing in memory shouldn't be too expensive unless the user has millions
  of completed apps in the history (at which point I'd expect other issues to arise
  aside from history server memory usage, such as FileSystem.listStatus()
  starting to become ridiculously expensive).

I also fixed a few minor things along the way which aren't really worth mentioning.
I also removed the app's log path from the UI since that information may not even
exist depending on which backend is used (even though there is only one now).
Right now there are two ways of setting the logging directory, one through the command line and the other through `spark.history.fs.logDirectory`. In the long run, we should favor the latter, as your comments seem to suggest. Given that, I think we should just remove all documentation of the command line way, but for backward compatibility keep the code that supports it. Otherwise, it's a little confusing for the user if we document both ways. (Note that there are multiple files that document this: HistoryServer.scala, HistoryServerArguments.scala, and monitoring.md)

Then,  in `start-history-server.sh`, if the user passes in a command line argument, we should print a deprecated warning, and convert that to `spark.history.fs.logDirectory` as a JVM `-D` option.