Replaced PartitionTextFileWriter with PartitioningDataStoreWriter - a mo...

...re generic implementation with pluggable entity writers and serialization formats.

Refactored the code so that writing, sharding and partitioning are separate concerns. 

The ShardedDataStoreWriter accepts a DataStoreWriterFactory to decouple it from the writing implementation.

The PartitioningDataStoreWriter accepts a ShardedDataStoreWriter, a PartitionStrategy and a Serializer which converts the entities to be written to the type accepted by the DataStoreWriters.

Some of the various *ObjectSupport methods have been made public so that they can be used from the DataStoreWriterFactory implementations (such as TextFileWriterFactory).
Serializer is used in PartitionDataStoreWriter and its default
implementation.

D.


P.S. The reason I did this is that I have a custom type called Event, which
I wanted to write to partitioned files using a TextFileWriter. So I wanted
my partitioned writer to have methods like

write(Event e) // write an Event and extract the partitioning key from it

write(Event e, Event key) // write an Event and use the event itself as the
partitioning key (why would I want to create e.g. a Map from the Event?
That wastes performance in terms of CPU as well as RAM)

But e.g. the existing PartitionTextFileWriter is <String, K>, so I'd have
to write a new implementation of AbstractPartitionDataStoreWriter to cope
with the entity type being Event, and then you have to implement more and
more...

D.

On Mon, Nov 24, 2014 at 3:19 PM, Janne Valkealahti <notifications@github.com
> wrote:

> In
> spring-hadoop-store/src/main/java/org/springframework/data/hadoop/store/Serializer.java:
>
> > @@ -0,0 +1,15 @@
> > +package org.springframework.data.hadoop.store;
> > +
> > +/**
> > + * Interface defining a serializer used to convert arbitrary entities to a format accepted by DataWriters
> > + *
> > + * @author mcintyred
> > + *
> > + * @param <E> the type of the entity to serialize
> > + * @param <S> the serialization format
> > + */
> > +public interface Serializer<E, S> {
>
> Where this is used?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/spring-projects/spring-hadoop/pull/185/files#r20795510>
> .
>



-- 
----
Duncan McIntyre
Associate Consultant
Pivotal Services@EMC EMEA
dmcintyre@gopivotal.com
+44 7917 580 118