aodn/harvesters — pull request #62 by xhoenner (2014-08-28 00:36:38+00:00)
Title: Ticked cursor size and specified batch 10000 in Populate_layers tPostgresqlInput_1

Addressed the issue listed below as Craig suggested, please review:

On 26 Aug 2014, at 1:47 pm, Craig Jones wrote:

I had a look at the log and it looks like the harvester is running out of memory in Populate_layers tPostgresqlInput_1. The harvester is currently reading all records into memory before handing them onto iPostgresqlOutput_1.

We have already assigned 4Gb of memory to this job so I don't think we should increase it. We need to be able to use the cursor option (on Advanced Settings for tPostgresqlInput_1) so that records are read in batches and fix it if there are any problems with this option. Can you make the required changes to the harvester/test please.