the astute reader will note the reinit of |tokens| in l. 145
(i split mandatory:foo=bar into mandatory and foo=bar, and then split foo=bar)