update tests; add preprocessing step to regularize delimiters