HPCC-10155 Prevent DFU Spray to output 100,000's of "CSV splitRecordSize(65536)
 at end of file" for hours.

This change implement buffer overruns monitoring. Every time when
getSplitRecordSize() reach at the end of the current buffer (and it is not
forced by processFullBuffer parameter) it counts the event and cumulate the
processed bytes. When the number of events exceeds the
MAX_NUMBER_OF_BUFFER_OVERRUN value it generates an exception to break this
possible wrong process.

This changes impacts both CCsvPartionier and CUtfPartitioner.

(This prevention are already implemented into corresponding QuickPartitioners)

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-10247 This should convert field names from the source unicode encoding
(e.g., could be utf16)

Add conversion for fieldnames defined in the first row.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-10271 Simplify selection between CCsvPartitoner and CCsvQuickPartioner.

Add code to check the source file is quoted or not at the partitioner creation
stage. If it is quoted then use CCsvPartitioner() else use
CCsvQuickPartitioner.

Remove unnecessary code from CCsvQuickPartitioner constructor and
CCsvPartitioner::findSplitPoint() method.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-10155 Prevent DFU Spray to output 100,000's of "CSV splitRecordSize(65536)
 at end of file" for hours.

Update after review.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-10247 This should convert field names from the source unicode encoding
(e.g., could be utf16)

Update after review.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-10155 Prevent DFU Spray to output 100,000's of "CSV splitRecordSize(65536)
 at end of file" for hours.

Update after review.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
Merge pull request #5034 from AttilaVamos/HPCC-10155-fix

HPCC-10155 Prevent DFU Spray to output 100,000's of "CSV splitRecordSize(65536) at end of file" for hours.

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
HPCC-10271 Simplify selection between CCsvPartitoner and CCsvQuickPartioner.

Update after review

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
Merge pull request #5040 from AttilaVamos/HPCC-10271-fix

HPCC-10271 Simplify selection between CCsvPartitoner and CCsvQuickPartioner.

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #5044 from AttilaVamos/HPCC-10247-improvement

HPCC-10247 This should convert field names from the source unicode encoding  (e.g., could be utf16)

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>