I think the content of the for loop can be simplified by calling "continue" early if there is no match. Like this:

    if not match:
        continue

This way the if and the elif can be simplified as there is no need to check for
the validity of `match` anymore.

I am also not sure why you test for match.groups()[0] not being None: in which situation can this happen?

If it cannot happen, then the code can be simplified like this (untested):

    if match.groups():
        tokenizer(self, match.group(1))
        self.idx = match.end(1)
    else:
        tokenizer(self, match.group(0))
        self.idx = match.end(0)
    return

Can you give these changes a try?