Package: tf-transformers
Tensorflow TransformersWebsite: https://legacyai.github.io/tf-transformerstf-transformers: faster and easier state-of-the-art Transformer in TensorFlow 2.0Imagine auto-regressive generation to be90xfaster.
tf-transformers (Tensorflow Transformers) is designed to harness the full power of Tensorflow 2, designed specifically for Transformer based architecture.These models can be applied on:📝 Text, for tasks like text classification, information extraction, question answering, summarization, translation, text generation, in over 100 languages.🖼️ Images, for tasks like image classification, object detection, and segmentation.🗣️ Audio, for tasks like speech recognition and audio classification. (Coming Soon)Unique FeaturesFaster AutoReggressive DecodingTFlitesupportCreating TFRecords is simple.Auto-Batching tf.data.datasetortf.raggedtensorsEverything is dictionary (inputs and outputs)Multiple mask modes likecausal,user-defined,prefix.tensorflow-text tokenizersupportSupports GPU, TPU, multi-GPU trainer with wandb, multiple callbacks, auto tensorboardBenchmark on GPT2 text generationGPT2 text generation withmax_length=64,num_beams=3.tf_transformers : 31 minutes
huggingface_tf  : 83 minutes
huggingface_pt  : 36 minutes
huggingface_jax : 35 minutesFrom83 minutesto31 minutesis a significant speedup.167 %1speedup.
On an average,tf-transformersis80-90 timesfaster thanHuggingFaceTensorflowimplementation and in most cases it iscomparableorfasterthanPyTorch.More benchmarks can be found inbenchmarkInstallationWith pipThis repository is tested on Python 3.7+ and TensorFlow 2.7.Recommended prerequistespipinstallsentencepiece
pipinstalltensorflow-text>=2.7.3
pipinstalltqdmInstalltensorflow >= 2.7.0 [CPU or GPU]as per your machine.
You should install tf-transformers in avirtual environment. If you're unfamiliar with Python virtual environments, check out theuser guide.First, create a virtual environment with the version of Python you're going to use and activate it.Then, you will need to install at least one of TensorFlow.
Please refer toTensorFlow installation page, installation pages regarding the specific install command for your platform. We highly recommend to install [tensorflow-text]
(https://www.tensorflow.org/text).When one of those backends has been installed, tf-transformers can be installed using pip as follows:pipinstalltf-transformersFrom sourcegitclonehttps://github.com/legacyai/tf-transformers.git
pipinstallpoetrycdtf-transformers
poetryinstallQuick tourtf-transformers API is very simple and minimalistic.>>>fromtf_transformers.modelsimportGPT2Model>>>model=GPT2Model.from_pretrained('gpt2')>>>model.save_checkpoint("/tmp/gpt2_model/")# Save ModelFor text-generation, it is very important to add :obj:use_auto_regressive=True. This is required for all the models.>>>fromtf_transformers.modelsimportGPT2Model>>>model=GPT2Model.from_pretrained('gpt2',use_auto_regressive=True)To serialize save and load model>>>fromtf_transformers.modelsimportGPT2Model>>>model=GPT2Model.from_pretrained('gpt2')>>>model.save_transformers_serialized("/tmp/gpt2_serialized/")# To load a serialized models for inference in prodcution:>>>importtensorflowastf>>>loaded=tf.saved_model.load("/tmp/gpt2_serialized/")>>>model=loaded.signatures['serving_default']TutorialsWe have covered tutorials covering pre-training, finetuning, classfication, QA, NER so much more.Read and Write TFRecords using tftText Classification using AlbertDynamic MLM (on the fly pre-processing using tf-text) in TPUImage Classification ViT multi GPU mirroredSentence Embedding train from scratch using Quoara on Roberta + Zeroshot STS-BPrompt Engineering using CLIPQuestion Answering as Generation - Squad v1 using GPT2Code to Code translation (CodexGLUE - Java to C#) using T5Model usageText Generation using GPT2Text Generation using T5Sentence TransformersTFlite TutorialsAlbert TFliteBert TFliteRoberta TFliteWhy should I use tf-transformers?Use state-of-the-art models in Production, with less than 10 lines of code.High performance models, better than all official Tensorflow based modelsVery simple classes for all downstream tasksComplete TFlite support for all tasks.Make industry based experience to avaliable to students and community with clear tutorialsTrain any model onGPU,multi-GPU,TPUwith amazingtf.keras.Model.fitTrain state-of-the-art models in few lines of code.All models are completely serializable.Customize any models or pipelines with minimal or no code change.ResearchTheResearchsection has codes
for pre-training different models ranging from **MLM, T5, CLIP etc **. All these scripts are designed
to harness full power of tensorflow-io pipeline and tested on TPU V2 and TPU V3. Bugs are expected in
those, but it serves as a purpose for practioners to start or modifying what we have already done.ContributionsJoint Albert(Smallest and best Transformer based model ever) on GLUE.We have conducted few experiments to squeeze the power ofAlbert basemodels ( concept is applicable to any models and in tf-transformers, it is out of the box.)The idea is minimize the loss for specified task in each layer of your model and check predictions at each layer. as per our experiments, we are able to get the best smaller model (thanks toAlbert), and fromlayer 4onwards we beat all the smaller model inGLUEbenchmark. Bylayer 6, we got aGLUEscore of81.0, which is4points ahead ofDistillbertwith GLUE score of77andMobileBertGLUE score of78.TheAlbertmodel has14 millionparameters, and by usinglayer 6, we were able to speed up the compuation by 50% .The concept is applicable to all the models and tasks.Codes + Read MoreLong Block Sequence TransformerBy splitting input sequence into block attention and merge using FFN layer we have shown that, smaller machines will be able to perform sequence processing up to 4096 tokens in a single V100 GPU machine.
The model has outperformsPegasus Base (128 million)inPubMedsummarisation despite being60 millionparameter.Codes + Read MoreSupported Models architecturestf-transformers currently provides the following architectures .ALBERT(from Google Research and the Toyota Technological Institute at Chicago) released with the paperALBERT: A Lite BERT for Self-supervised Learning of Language Representations, by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.BERT(from Google) released with the paperBERT: Pre-training of Deep Bidirectional Transformers for Language Understandingby Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.BERT For Sequence Generation(from Google) released with the paperLeveraging Pre-trained Checkpoints for Sequence Generation Tasksby Sascha Rothe, Shashi Narayan, Aliaksei Severyn.ELECTRA(from Google Research/Stanford University) released with the paperELECTRA: Pre-training text encoders as discriminators rather than generatorsby Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.GPT-2(from OpenAI) released with the paperLanguage Models are Unsupervised Multitask Learnersby Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.MT5(from Google AI) released with the papermT5: A massively multilingual pre-trained text-to-text transformerby Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel.RoBERTa(from Facebook), released together with the paper aRobustly Optimized BERT Pretraining Approachby Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.T5(from Google AI) released with the paperExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformerby Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.Vision Transformer (ViT)(from Google AI) released with the paperAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scaleby Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby.
10CLIP(from OpenAI) released with the paperLearning Transferable Visual Models From Natural Language Supervisionby Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever.CitationWe now have apageyou can cite for the tf-transformers library.