mesos/spark — pull request #713 by JoshRosen (2013-07-18 06:16:50+00:00)
Title: Add JavaAPICompletenessChecker script to find methods that are missing from the Java API

This pull request adds a script for automatically identifying methods in Spark's Scala API that need to be ported to its Java API. The script, JavaAPICompletenessChecker, works by enumerating the public methods in the Scala API, converting the method signatures to their Java equivalents using a set of rewrite rules, and searching the Java API for those methods. For each missing method, the tool prints its expected Java signature.

This technique isn't perfect, but it does a pretty good job of finding missing methods or methods with signatures that are inconsistent with the Java API conventions. For example, check out the [list of methods missing from the 0.7.3 Java API](https://gist.github.com/JoshRosen/6026985).

In order to complete the Java API in both 0.7 and master, I'm submitting this pull request against branch-0.7. My plan is to first add the missing methods in 0.7, then cherry-pick those commits into master and add the missing methods from master.

I created a new `tools` subproject to hold this tool. I considered putting it in `examples`, but I thought that would be confusing. It would be possible to store this tool in its own Scala / Maven project that adds Spark as a dependency, but including it in Spark proper makes it easier to iteratively re-run the tool as you port methods to the Java API.

One caveat about this tool: for Java API methods that are overloaded on `spark.java.api.function.*` types, the tool only reports the non-specialized methods; implementors will still need to implement the additional methods for PairFunction, DoubleFunction, etc. Similarly, it doesn't catch methods that need to be re-implemented in JavaDoubleRDD, JavaPairRDD, and JavaRDD, so it won't catch mistakes like implementing same-result-type methods like `map` in JavaRDD but forgetting to implement them elsewhere. Both of these limitations are fixable, but they'd add a lot of complexity and I think the current code is still useful as-is (I may fix this later, but I'm a bit tired of working on this for now).