minor, generate n-grams by copying portions from the output buffer directly

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
minor, do away with gram string caching, just keep an accumulator

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
hygiene, also force-write the feed line

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
Merge branch 'tool/terms'

* tool/terms:
  hygiene, also force-write the feed line
  minor, do away with gram string caching, just keep an accumulator
  minor, generate n-grams by copying portions from the output buffer directly
minor, unroll upper->lower map completely in --bitfields mode in gencls(1)

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
minor, go for a sparser representation of genmap[], using genmof[] to indicate offsets

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
minor, use upper->lower bitfield maps

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
test, provide regressions for utf8-heavy inputs

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
minor, carry feeds through to the output

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
fix, look up upper->lower maps at offset MOF

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
fix, PEBKAC, access genmap3 maps at offset NX2

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
minor, thoroughly inspect streak buffer in --normal-form mode for lower case characters

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
test, fix, single captial letters remain ucased in --normal-form mode

Signed-off-by: Sebastian Freundt <freundt@ga-group.nl>
Merge branch 'tool/terms'

* tool/terms:
  test, fix, single captial letters remain ucased in --normal-form mode
  minor, thoroughly inspect streak buffer in --normal-form mode for lower case characters
  fix, PEBKAC, access genmap3 maps at offset NX2
  fix, look up upper->lower maps at offset MOF
  minor, carry feeds through to the output
  test, provide regressions for utf8-heavy inputs
  minor, use upper->lower bitfield maps
  minor, go for a sparser representation of genmap[], using genmof[] to indicate offsets
  minor, unroll upper->lower map completely in --bitfields mode in gencls(1)