map-side lots of SSOnes are added to SSMany

reduce side its simply a matter of merging the SSManys, which depending on
your cluster side isnt that many. this is where the wasteful re-calculation
of bucketsOption is done. my experience is that this reduce side step is
very fast. the work sits all map-side. so i didnt put any effort into
optimizing this.

sure we can make it RT serialization. choices are
1) making bucketsOption an implementation detail by making it private and
changing equals and print and all that good stuff, or
2) serializing bucketsOption and opening up the constructor that can lead
to invalid state if someone misuses it


On Mon, Mar 10, 2014 at 1:41 PM, ianoc <notifications@github.com> wrote:

> In algebird-core/src/main/scala/com/twitter/algebird/SpaceSaver.scala:
>
> > @@ -101,7 +101,7 @@ object SSMany {
> >    private def apply[T](capacity: Int, counters: Map[T, (Long, Long)], buckets: SortedMap[Long, Set[T]]): SSMany[T] =
> >      SSMany(capacity, counters, Some(buckets))
> >
> > -  private def apply[T](capacity: Int, counters: Map[T, (Long, Long)]): SSMany[T] =
> > +  def apply[T](capacity: Int, counters: Map[T, (Long, Long)]): SSMany[T] =
>
> So i think being able to safely/sanely RT any structure is critical to
> ensuring our serialization paths are robust.
>
> How you mean though that it only comes up during reduce? If we
> pre-aggregate then we will have data in these fields we will need to
> re-calculate?
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/twitter/algebird/pull/289/files#r10440404>
> .
>