spring-projects/spring-hadoop — pull request #92 by jvalkeal (2014-05-14 18:43:39+00:00)
Title: SHDP-334 Support for store writer partitioning

- New PartitionDataStoreWriter and PartitionTextFileWriter
- Adding concepts of partition key and resolvers
- Better handling of spel expressions
- Partition writer tweaks to handle all necessary
spring lifecycle and other needed components
because additional writers wrapped by a partition
writer are not handled by app context.
- New UuidFileNamingStrategy to help with naming issues
- Writer base classes now have better support
for using partitioned paths.
- Naming strategies how have better tests
- Changed FileNamingStrategy to separate
next() and reset().

Squashed commit of the following:

commit ac9b7369bd080f04642a64fea5fcfa6079fbac32
Author: Janne Valkealahti
Date: Wed May 14 19:17:58 2014 +0100

SHDP-334 Remove unused import

commit 9c6480c888d77cafec5d873ae761e5ab5a6678a6
Author: Rodrigo Meneses
Date: Wed May 14 09:53:00 2014 -0700

SHDP-334. fix bug on DateFormatMethodExecutor

commit 59db45b14c461c21fcef2ae9ce1f502b906ff33b
Author: Rodrigo Meneses
Date: Tue May 13 16:30:03 2014 -0700

SHDP-334 - Enhance dateFormat internal function to accepts dates with
String representation or using an actual Date object

commit a71699939a2a7f5fa08e06d6ce1695ff2720b7d8
Author: Janne Valkealahti
Date: Tue May 13 17:50:10 2014 +0100

SHDP-334 Remove PartitionKey wrapper

- Instead of wrapping generic object within
a PartitionKey, just use K partitionKey
and let all other components follow same logic.

commit 5d01eba468e59dd63ae2da37f74dcfc009ec1475
Author: Janne Valkealahti
Date: Tue May 13 16:53:46 2014 +0100

SHDP-334 Fix naming with uuid

- Changed FileNamingStrategy to separate
next() and reset().
- Fixed bug which got wrong rolling part especially
when used with uuid in file name and with multiple writers.

commit 1d95df2666664787751d0e4ea480e3b0d4b20fe2
Author: Janne Valkealahti
Date: Tue May 13 11:30:58 2014 +0100

SHDP-334 Refactor generic partition keys

- Generic which used a Map now uses a Spring
Message which allows to evaluate payload
together with headers.
- Spel context is now cached so that it wont
be re-created with every data write. At a same
time spel variable registration were removed to
be able to use same context. 'headers', 'payload'
and header fields can still be accessed with
shortcuts.

commit 48a9a8e79c72097a98dd6e047e862f0e60b75abf
Author: Janne Valkealahti
Date: Mon May 12 13:41:38 2014 +0100

SHDP-334 Better support for partitioning

- New UuidFileNamingStrategy to help with naming issues
- Writer base classes now have better support
for using partitioned paths.
- Naming strategies how have better tests

commit 910187e8d32866052aaa6e9c904184237e563c4b
Author: Janne Valkealahti
Date: Fri May 9 09:30:23 2014 +0100

SHDP-334 Rename AbstractDataStorePartitionWriter

- To follow naming logic, renaming AbstractDataStorePartitionWriter
to AbstractPartitionDataStoreWriter. Also TextFilePartitionedWriterTests
named to PartitionTextFileWriterTests.

commit 6bdf1c1550c5fff2976342b809dcdcffd1a96cf6
Author: Janne Valkealahti
Date: Thu May 8 17:00:55 2014 +0100

SHDP-334 Additional work for partitioning

- Adding concepts of partition key and resolvers
- Better handling of spel expressions
- Partition writer tweaks to handle all necessary
spring lifecycle and other needed components
because additional writers wrapped by a partition
writer are not handled by app context.

commit 1139e70b7c8e7ce09205040f7ad5995b6701fb83
Merge: 7cea938 f02cb3b
Author: Janne Valkealahti
Date: Thu May 8 16:47:44 2014 +0100

Merge remote-tracking branch 'rmenesespiv/SHDP-334' into SHDP-334-merge-work

commit f02cb3bd936feb8be5112a436596b8c18de5504e
Author: Rodrigo Meneses
Date: Tue May 6 13:48:45 2014 -0700

SHDP-334
PartitionTextFileWriter will now create bean strategies (fileName + rollOver) using prototype scope, since each TextFileWriter needs its own strategies to work independently of each other

commit bf1d9647cce7c90cc07e362120642f01ac1e3580
Author: Rodrigo Meneses
Date: Fri May 2 15:04:52 2014 -0700

SHDP-334
Propagate file naming strategy from PartitionWriter to its internal
writers

commit cbd7bae8113e7472c758ed3a1720e307726bc4aa
Author: Rodrigo Meneses
Date: Fri May 2 09:33:36 2014 -0700

SHDP-334
change directory to partition in all new classes/methods
create a TextFileWriterFactoryBean so that the delegaate workers in
Partition are managed by spring context

commit 9c4b08074ab0e387a57de6ac9f39b2ce09814515
Author: Rodrigo Meneses
Date: Wed Apr 30 16:46:06 2014 -0700

SHDP-334
Define a new interface PartitionDataStoreWriter exposing a
writeToPartition method which implementing classes needs to define to
support writing to a given partition