not sure about SerializeStream(). it will break the ObjectOutput envelope
imo.


On Tue, Jul 30, 2013 at 6:38 PM, Matei Zaharia <notifications@github.com>wrote:

> In core/src/main/scala/spark/rdd/ParallelCollectionRDD.scala:
>
> > +    val sfactory = SparkEnv.get.serializer
> > +    // treat java serializer with default action rather
> > +    // than going thru serialization,
> > +    // to avoid a separate serialization header.
> > +    sfactory match {
> > +      case js: JavaSerializer => out.defaultWriteObject()
> > +      case _ => {
> > +        // for every other serializer, we
> > +        // assume that it would support Seq[T] and
> > +        // do so efficiently.
> > +        val ser = sfactory.newInstance()
> > +        out.writeLong(rddId)
> > +        out.writeInt(slice)
> > +        val bb = ser.serialize(values)
> > +        out.writeInt(bb.remaining())
> > +        Utils.writeByteBuffer(bb, out)
>
> In here, use the serializeStream method of spark.Serializer instead of
> creating a giant ByteBuffer. It will probably be more efficient for Seq
> types by serializing each object one by one.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/mesos/spark/pull/724/files#r5493991>
> .
>