Extracted tokens() function and added it to Collation class.
Expanded LCPInterval class with functionality from LCPSubinterval class.
Moved split_into_smaller_intervals() function from Collation class to
LCPSubInterval class. Expanded the implementation of
split_into_smaller_intervals() function to also handle descending LCP
intervals.