Encode.pm change to the UTF8 flag

In the past, decode_utf8() silently ignored attempts to decode data
with the UTF8 flag on. However, Encode.pm changed around 2.53 and
resulted in wide-character errors preventing the likes of
--load-all-sample from working.

Since our data should already be UTF8 when it hits these functions, we
should not be trying to decode it in the first place.

Signed-off-by: Dan Wells <dbw2@calvin.edu>
Signed-off-by: Dan Scott <dscott@laurentian.ca>
Test Perl Unicode normalization process

Add some basic tests of our Unicode normalization (and entityization)
process.

Also, 'use utf8;' tells the script that it contains UTF8 characters,
not binary bytes, and resolves a test problem.

Signed-off-by: Dan Wells <dbw2@calvin.edu>
Signed-off-by: Dan Scott <dscott@laurentian.ca>
Add pgTAP test for normalized MARC records

Specifically, we're concerned about ensuring that any changes related to
the fix for Encode (requiring the removal of the effectively useless but
now erroring decode_utf8()) do not inadvertently change the content of
the MARC records on ingest. Because we wouldn't want to have to reingest
everything to normalize search, etc.

Signed-off-by: Dan Scott <dscott@laurentian.ca>
Signed-off-by: Dan Wells <dbw2@calvin.edu>
Upgrade file for Encode.pm 2.54+ changes

Signed-off-by: Dan Wells <dbw2@calvin.edu>
Signed-off-by: Remington Steed <rjs7@calvin.edu>
Stamping 0851 - changes for Encode.pm 2.54+

Signed-off-by: Dan Wells <dbw2@calvin.edu>