SpringSource/spring-batch — pull request #147 by sshcherbakov (2013-03-20 14:00:10+00:00)
Title: BATCH-1613: MultiThreadedFlatFileItemReader and FlatFilePartitioner impl...

...ementations.

Recently, we have faced a need to prepare a solution for speeding up Spring Batch file reading process. Since the standard FlatFileItemReader implementation is not intended for use in a multi-threaded environment, we have looked at a possibly existing solution and found the discussion on Spring Batch forum:
http://forum.springsource.org/showthread.php?93151-multi-threaded-FlatFileItemReader
and the https://jira.springsource.org/browse/BATCH-1613 Jira issue.
Basing on the code provided by Stephane Nicoll in the forum I have prepared a pull request for your consideration.
The MultiThreadedFlatFileItemReader and FlatFilePartitioner have been reworked in order to improve performance
Below you can see the performance comparison on a 8 core MacBook Pro with an SSD drive.
The test (which can be found in the pull request) only reads items from a temporary flat file.
Though applying multiple threads for reading does not bring much in terms of the reading performance, as can be seen below, it can be useful when processing or writing step is blocking on an intensive I/O. The test simulated 10ms delay by simply calling Thread.sleep(10).

The improvements to the initial implementation include:
- Underlying file system seek function utilized in order to locate the starting point for a reading thread, so that partition processing threads do not need to read forward to the starting point;
- The partitioning occurs at the new line borders on the same first fast reading sweep over the file (measured 2 to 5 times faster performance in comparison to BufferedReader and LineNumberReader implementation); further improvements like support of multi-line records possible;
- Partitioner distributes data between partitions more evenly to avoid having a partition twice as large as the others in an unlucky case
- Some unit tests added

Performance measured on a MacBook Pro i7 8 cores with SSD drive and on
a Intel Q6600 2,4GHz 4 cores CPU with ordinary HDD.
Input file 15232 lines (4Mb+)
Results are in milliseconds average of 4 attempts

Current implementation:
-----------------------
Cores	Threads	No delay (log off); No delay (log DEBUG); Delay 10ms (log DEBUG);

8	1	6003	7595	171924
2	3531	5909	86172
4	3154	5562	43291
8	3329	5583	21979
16	3624	5670	11492

4	1	8554	10734	188855
2	5402	8235	98238
4	5093	8057	44679
8	5115	8398	21623
16	5318	8389	11990

Initial implementation (submitted in the forum):
------------------------------------------------
Cores	Threads	No delay (log off); No delay (log DEBUG); Delay 10ms (log DEBUG);

8	1	6063	9656	172209
2	4420	9138	87852
4	4579	9833	45966
8	6193	11680	26536
16	9248	20492