Package: tfaip
tfaip- A Generic and Powerful Research Framework for Deep Learning based on Tensorflowtfaipis a Python-based research framework for developing, organizing, and deploying Deep Learning models powered byTensorflow.
It enables to implement both simple and complex scenarios that are structured and highly configurable by parameters that can directly be modified by the command line (read thedocs).
For example, thetutorial.full-scenario for learning MNIST allows to modify the graph during training but also other hyper-parameters such as the optimizer:exportPYTHONPATH=$PWD# set the PYTHONPATH so that the examples dir is found# Change the graphtfaip-trainexamples.tutorial.full--model.graphMLP--model.graph.nodes20010050--model.graph.activationrelu
tfaip-trainexamples.tutorial.full--model.graphMLP--model.graph.nodes20010050--model.graph.activationtanh
tfaip-trainexamples.tutorial.full--model.graphCNN--model.graph.filters4020--model.graph.dense100# Change the optimizertfaip-trainexamples.tutorial.full--trainer.optimizerRMSprop--trainer.optimizer.beta10.01--trainer.optimizer.clip_global_norm1# ...A trained model can then easily be integrated in a workflow to predict provideddata:predictor=TutorialScenario.create_predictor("PATH_TO_TRAINED_MODEL",PredictorParams())forsampleinpredictor.predict(data):print(sample.outputs)In practice,tfaipfollows the rules of object orientation, i.e., the code for a scenario (e.g., image-classification (MNIST), text recognition, NLP, etc.) is organized by implementing classes.
By default, eachScenariomust implementModel, andData.
Seeherefor the complete code to run the upper example for MNIST and seeherefor the minimal setup.SetupTo setuptfaipcreate a virtual Python (at least 3.7) environment and install thetfaippip package:pip install tfaip:virtualenv-ppython3venvsourcevenv/bin/activate
pipinstalltfaip
pipinstalltfaip[devel]# to install additional development/test requirementsHave a look at thewikifor further setup instructions.Run the TutorialAfter the setup succeeded, launch a training of the tutorial which is an implementation of the common MNIST scenario:exportPYTHONPATH=$PWD# set the PYTHONPATH so that the examples dir is foundtfaip-trainexamples.tutorial.full# If you have a GPU, select it by specifying its IDtfaip-trainexamples.tutorial.full--device.gpus0Next StepsStart reading theMinimum Tutorial, optionally have a look at theFull Tutorialto see more features.
Thedocsprovides a full description oftfaip.To set up anew custom scenario, copy thegeneral templateand implement the abstract methods.
Consider renaming the classes!
Launch the training by providing the path or package-name of the new scenario whichmustbe located in thePYTHONPATH!Features oftfaiptfaipprovides different features which allow designing generic scenarios with maximum flexibility and high performance.Code designFully Object-Oriented: Implement classes and abstract functions or overwrite any function to extend, adapt, or modify its default functionality.Typing support:tfaipis fully typed with simplifies working with an IDE (e.g., use PyCharm!).Using pythonsdataclassesmodule to set up parameters which are automatically converted to parameters of the command line by ourpaiargparsepackage.Data-PipelineEvery scenario requires the setup of a data-pipeline to read and transform data.tfaipoffers to easily implement and modify even complex pipelines by defining multipleDataProcessorswhich usually implement a small operation to map an input sample to an output sample.
E.g., oneDataProcessorloads the data (input=filename,output=image), another one applies normalization rules, again another one applies data augmentation, etc.
Thegreat advantageof this setup is that the data processors run in Python and can automatically be parallelized bytfaipfor speed up by settingrun_parallel=True.Deep-Learning-FeaturesSincetfaipis based on Tensorflow the full API are available for designing models, graphs, and even data pipelines.
Furthermore,tfaipsupports additional common techniques for improving the performance of a Deep-Learning model out of the box:Warm-starting (i.e., loading a pretrained model)EMA-weightsEarly-StoppingWeight-Decayvarious optimizers and learning-rate schedulesContributingWe highly encourage users to contribute own scenarios and improvements oftfaip.
Please read thecontribution guidelines.BenchmarksAll timings were obtained on a Intel Core i7, 10th Gen CPU.MNISTThe following Table compares the MNIST Tutorial of Keras to theMinimum Tutorial.
The keras code was adopted to use the same network architecture and hyperparemter settings (batch size of 16, 10 epochs of training).CodeTime Per EpochTrain AccVal AccBest Val AccKeras16 s99.65%98.24%98.60%tfaip18 s99.76%98.66%98.66%tfaipand Keras result in comparable accuracies, as to be expected since the actual code for training the graph is fundamentally identical.tfaipis however a bit slower due some overhead in the input pipeline and additional functionality (e.g., benchmarks, or automatic tracking of the best model).
This overhead is negligible for almost any real-world scenario because due to a clearly larger network architecture, the computation times for inference and backpropagation become the bottleneck.Data PipelineIntegrating pure-python operations (e.g., numpy) into atf.data.Datasetto apply high-level preprocessing is slow by default sincetf.data.Dataset.mapin cooperation withtf.py_functiondoes not run in parallel and is therefore blocked by Python's GIL.tfaipcurcumvents this issue by providing an (optional) parallelizable input pipeline.
The following table shows the time in seconds for two different tasks:PYTHON: applying some pure python functions on the dataNUMPY: applying several numpy operations on the dataModeTaskThreads 1Threads 2Threads 4Threads 6tf.py_functionPYTHON23.4722.7824.3825.76tfaipPYTHON26.6814.488.118.13tf.py_functionNUMPY104.1082.7876.3377.56tfaipNUMPY97.0756.9343.7842.73The PYTHON task clearly shows thattf.data.Dataset.mapis not able to utilize multiple threads.
The speed-up in the NUMPY tasks occurs possibly due to paralization in the numpy API to C.