Feature/filter unchanged processor

Performance and necessary storage space will be (slightly) better when searching/comparing. Unless we decide to switch over to a checksum algorithm with a larger output, it should not be a problem.

From: Jeffrey Licht [mailto:notifications@github.com]
Sent: Wednesday, November 19, 2014 16:04
To: harvard-library/librarycloud_ingest
Cc: Renaud Holcombe
Subject: Re: [librarycloud_ingest] Feature/filter unchanged processor (#25)


In src/main/java/edu/harvard/libcomm/pipeline/FilterUnchangedProcessor.java:

> +            log.info("starting insert!");

> +            Set<String> duplicateRecordIds = ingestDao.CheckAndSaveItemChecksum(recordMap);

> +            log.info(duplicateRecordIds.size() + " duplicate records found.");

> +

> +            if(duplicateRecordIds == null ||  duplicateRecordIds.size() == 0){

> +             return;

> +            }

> +            //remove duplicates

> +            String duplicateRecordIdString = "<recordIdList><recordId>" + StringUtils.join(duplicateRecordIds,"</recordId><recordId>") + "</recordId></recordIdList>";

> +            String scrubbedMessage = MessageUtils.transformPayloadData(libCommMessage, "src/main/resources/Remove-MODS-Records.xsl", duplicateRecordIdString);

> +

> +        payload.setData(scrubbedMessage);

> +        libCommMessage.setPayload(payload);

> +     }

> +

> +     private Map<String, Integer> SplitIdAndChecksum(String body)

Is there any reason not to treat the checksum as a string?

—
Reply to this email directly or view it on GitHub<https://github.com/harvard-library/librarycloud_ingest/pull/25/files#r20607333>.