updated Makefile
made function to sum segment sizes
fixed tokenizers

questioning their usefulness now
updated tests