DaveChild/Text-Statistics — pull request #7 by DrDub (2012-12-07 10:55:55+00:00)
Title: Test cases derived from a pronunciation dictionary

Hi!

I tested your syllable counter with a 100k+ pronunciation dictionary
(available for example here:
http://packages.debian.org/unstable/sound/festlex-cmu)

Text-Statistics is getting 90% of the words right, congrats! I
packaged the ones wrong in a separate text case, with some luck they
can be fixed over time.

Also, I don't know where are you using the list of homographs with
different syllable count, but using the same data I found a few more:

aged
("aged" j (((ey jh) 1) ((ih d) 0)))
("aged" v (((ey jh d) 1)))

blessed
("blessed" j (((b l eh s) 1) ((ih d) 0)))
("blessed" v (((b l eh s t) 1)))

separate
("separate" n (((s eh p) 1) ((r ax t) 0)))
("separate" v (((s eh p) 1) ((er) 0) ((ey t) 1)))

shell one-liners used:

To generate the test cases I used some perl-foo:

cat /usr/share/festival/dicts/cmu/cmudict-0.4.out | \
tail -n +2 | \
perl -e 'use Data::SExpression; $ds=Data::SExpression->new; while(){$d=$ds->read($_); next if($d->[0] eq $prev); $prev=$d->[0]; $prev_s=scalar(@{$d->[2]}); print "if(\$s->syllable_count(\"$prev\") != $prev_s){ echo \"$prev\\n\"; };\n"}' >> ./massive-test.php

With a massive-test.php starting with
new; while(){ $d=$ds->read($_); if($d->[0] eq $prev && scalar(@{$d->[2]}) != $prev_s){ print "$d->[0]\n" }; $prev=$d->[0]; $prev_s=scalar(@{$d->[2]}); }'