Functional test for short token elimination