Create a new branch for 5.0.6

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-12633 Improve contention cost in splitter

In cases where the consumers keep pace and are waiting on the
produce side of the splitter, the cost of waiting was costly.
The change here, introduces a callback mechanism into the shared
write ahead, such that the producer [optionally] only triggers
the readers that there is something to consumer per page flush.
In the memory variety of the shared writer which blocks when it
is far enough ahead, the producer will write (and page) as fast
as possible on a async thread, and block when full.
All of this savs the cosumers, entering crits, waiting on
semaphores and waking up and re-entering crits etc. on a per row
basis when they are consuming quickly.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-12633 Some changes following review

1) Get rid of writer thread, no real benefit
2) Move readerWait into parent

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-12633 Add some memory buffering to spilling variety

Add a memory pool of a few chunks to disk write too.
This substantially speeds up splitter that can spill, when the
arms are actually close to each other.

Also change QueueOf::dequeue(element) to return element.
It makes it a little cleaner when disposing of a link counted
item.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-9917 Add webservice options to ECL

Add new FORMAT specifiers to STORED to affect field formatting and
sequence.

Add new #WEBSERVICE function for options affecting webservice.
Support explicit selection and ordering of specific fields.  Support
adding help and descriptive text.

Usage:

unsigned2 u1 := 0 : stored('u1', few, format(fieldwidth(8), sequence(12)));
integer2 i1 := 0 : stored('i11, format(fieldwidth(9), sequence(11)));

description("does cool stuff"));

Signed-off-by: Anthony Fishbeck <anthony.fishbeck@lexisnexis.com>
HPCC-12677 Allow max CSV row size to be configured

The max CSV row size (default 10MB) prevents unbound CSV lines
causing problems (e.g. consuming all of memory).
10MB should be sufficient in 99% of cases, but sometimes a higher
limit is required, so allow the hard-limit to be configured via
a query #option('maxCsvRowSize', <MB>)

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-11951 Update docs to say Roxie can write files

Signed-off-by: Jim DeFabia <jamesdefabia@lexisnexis.com>
Merge pull request #6750 from JamesDeFabia/roxiewrites

HPCC-11951 Update docs to say Roxie can write files

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-12743 Fix typo in DENORMALIZE example

Signed-off-by: Jim DeFabia <jamesdefabia@lexisnexis.com>
Merge pull request #6761 from JamesDeFabia/FixTypoNormalize

HPCC-12743 Fix typo in DENORMALIZE example

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-12677 Pickup option in Roxie too

Also
1) Move a couple of common options to thorcommon.hpp
2) Change outputLimit to outputLimitMb,
but continue to accept former too.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #6742 from afishbeck/webserviceEcl

HPCC-9917 Add webservice options to ECL

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #6741 from jakesmith/hpcc-12633

HPCC-12633 Improve contention cost in splitter

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
HPCC-12712 ECL CLI support for generating debug workunit shared objects

ecl publish, deploy, and run support for "-g" or "--debug" to generate
debug executable objects.

Signed-off-by: Anthony Fishbeck <anthony.fishbeck@lexisnexis.com>
Merge pull request #6770 from afishbeck/publishDebug

HPCC-12712 ECL CLI support for generating debug workunit shared objects

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
HPCC-12677 Fixed wrong static defaults used in a couple of places

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-12759 Fix incorrect test (mostly harmless)

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Merge pull request #6745 from jakesmith/hpcc-12677

HPCC-12677 Allow max CSV row size to be configured

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
HPCC-12761 Fix memcachedtest xml key

Signed-off-by: James Noss <james.noss@lexisnexis.com>
Merge pull request #6773 from ghalliday/issue12759

HPCC-12759 Fix incorrect test (mostly harmless)

Reviewed-By: Jake Smith <jake.smith@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>