Only clean up token queue when switching input files