Project: talenddataquality
Issue TDQ-8830: Create a tBatchAddressRowCloud component
We first need to enrich the Address API to support batch file processing. The process has several steps: # read data from input stream and prepare files (we may need several files during a stream) to be sent to the address provider. # send the file(s) to the address provider and get the results. # output the results in the output flow of the component The input stream may be splitted in several files (for instance, if there is a limit to the number of rows to submit to the address provider). So, there may be the need for a size parameter in the component. Each file can be sent independently. This means we don't need to wait for the response to the first file in order to start requesting the an address correction on the second file. We can even think to process data in parallel: when a Talend flow is parallelized in the job editor, several files of input data must be created. Each of the file must be sent as soon as ready to the Rest API. We must also be able to deactivate the parallel execution when the provider does not support several requests at the same time. Once the API works and Junit are created, we can develop the tBatchAddressRowCloud component. h1. Handle Loqate first This component receives address records and call the loqate batch API: https://www.everythinglocation.com/api-rest-batch/ See also https://www.everythinglocation.com/batch/list.php This component sends a file (formatted as expected by Loqate) to the web service, retrieves the corrected addresses and output them in an output flow.
Story points: 3