text: 241,26,1408469060052946951,790033937531703296,Yi Tay,"Excited to share our new work from @GoogleAI and @DeepMind. ""Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (paper: <LINK>) <LINK> We introduce a new inductive bias that learns latent subwords in an e2e fashion. Charformer is fast, often much faster than other byte-level baselines/subword models while achieving very competitive performance. No more building SPMs / re-pretraining for every new task/domain! Charformer works well both on monolingual English standard benchmarks (GLUE) and also multilingual datasets. We also evaluate Charformer on Jigsaw's toxicity detection sets from social media. Joint work with amazing collaborators. @vqctran (co-first author), @seb_ruder (DeepMind), @_jai_gupta @hwchung27 @dara_bahri @pierceqin @sens3 @congyu and @metzlerd. Code link is already in the paper but release ETA should be in about 1 week. :) @christopher Mesh Tensorflow, similar to the T5 library. We're also working on a JAX version which will be released sometime later.",https://arxiv.org/abs/2106.12672,"State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level. Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end. ","Charformer: Fast Character Transformers via Gradient-based Subword