Be less clever about tracking characters to process