flattening of tokens done