I would look at this as a very uncommon and temporary situation.
We would need to continue supporting the old api for only a little while
longer - after which we can remove the reflection stuff; and ideally, yarn
should converge on a stable api soon enough.
So until then, unfortunately, we have instability in our impl.


On Sat, Mar 15, 2014 at 10:32 PM, gzm55 <notifications@github.com> wrote:

> In
> core/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandlerMacro.scala:
>
> > +   * so we don't need to call getAMResponse(), just use AllocateResponse directly.
> > +   * This macro will test the existence of AMResponse,
> > +   * and generate diffenert expressions.
> > +   *
> > +   * This macro now is only used in spark's alpha version of yarn api.
> > +   * It stays in the core project, for the two-stage compiling of
> > +   * the scala macro system.
> > +   */
> > +  def getAMRespImpl(c: Context)(resp: c.Expr[Any]) = {
> > +    try {
> > +      import c.universe._
> > +      c.mirror.staticClass("org.apache.hadoop.yarn.api.records.AMResponse")
> > +      c.Expr[Any](Apply(Select(resp.tree, newTermName("getAMResponse")), List()))
> > +    } catch {
> > +      case _: Throwable => resp
> > +    }
>
> I tried to use runtime refleciton, and found some difficults. because
> amResp will have different type, I've to give it a duck type like
>
> AnyRef {
> def getAllocatedContainers ...
> def getAvailableResources ...
> def getCompletedContainersStatuses ...
> }
>
> or have to cache four reflection methods (the upper three plus
> getAMResponse()). in the future, each time we introduce a method of
> AMResponse/AllocateResponse, we'll have to make additional cache for it.
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/151/files#r10638121>
> .
>