Package: mann
MANNMANN, which stands for Multitask Artificial Neural Networks, is a Python package which enables creating sparse multitask models compatible withTensorFlow. This package contains custom layers and utilities to facilitate the training and optimization of models using the Reduction of Sub-Network Neuroplasticity (RSN2) training procedure developed byAI Squared, Inc.Changing to BeyondMLPlease be advised that the MANN package is going to be deprecated after version 0.3.0 and will be moved to thebeyondmlpackage in PyPi. For future releases, please alter your code to utilize thebeyondmlpackage.InstallationThis package is available throughPyPiand can be installed via the following command:pipinstallmannTo install the current version directly fromGitHubwithout cloning, run the following command:pipinstallgit+https://github.com/AISquaredInc/mann.gitAlternatively, you can install the package by cloning the repository fromGitHubusing the following commands:# clone the repository and cd into itgitclonehttps://github.com/AISquaredInc/manncdmann# install the packagepipinstall.Mac M1 UsersFor those with a Mac with the M1 processor, this package can be installed, but the standard version of TensorFlow is not compatible with the M1 SOC. In order to install a compatible version of TensorFlow, please install theMiniforgeconda environment, which utilizes the conda-forge channel only. Once you are using Miniforge, using conda to install TensorFlow in that environment should install the correct version. After installing TensorFlow, the commandpip install mannwill install the MANN package.ContributingFor those who are interested in contributing to this project, we first thank you for your interest! Please refer to the CONTRIBUTING.md file in this repository for information about best practices for how to contribute.Vulnerability reportingIn the event you notice a vulnerability within this project, please open aGitHub Issuedetailing the vulnerability to report it. In the event you would like to keep the report private, please emailmann@squared.ai.CapabilitiesThe MANN package includes two subpackages, themann.utilspackage and themann.layerspackage. As the name implies, themann.utilspackage includes utilities which assist in model training. Themann.layerspackage includes custom Keras-compatible layers which can be used to train sparse multitask models.UtilsThemann.utilssubpackage contains helper functions for performing training and conversion of models using masking layers.In addition to the functions just mentioned, there is also anActiveSparsificationcallback object which enables active sparsification during training rather than solely one-shot sparsification. Note that this callback currently only supports simultaneous training. We are working to support iterative training with this callback as well.mask_modelThemask_modelfunction is central to the RSN2 training procedure and enables masking/pruning a model so a large percentage of the weights are inactive.Inputs to themask_modelfunction are a TensorFlow model, a percentile in integer form, a method - either one of 'gradients' or 'magnitude', input data, and target data.get_custom_objectsTheget_custom_objectsfunction takes no parameters and returns a dictionary of all custom objects required to load a model trained using this package.remove_layer_masksTheremove_layer_masksfunction takes a trained model with masked layers and converts it to a model without masking layers.add_layer_masksTheadd_layer_masksfunction takes an existing model that has non-MANN layers and converts it so that all layers which have an analog in the MANN package. This enables pretrained models to be converted and sparsified.quantize_modelThequantize_modelfunction takes in a model and a datatype to quantize the model to.build_transformer_blockThebuild_transformer_blockfunction can be used to build a block in a transformer architecture.build_token_position_embeddingThebuild_token_position_embeddingfunction can be used to build a token and position embedding block for use in a transformer architecture model.Theget_task_masking_gradientsfunction retrieves the gradients of masking weights within a model.Themask_task_weightsfunction masks specific task weights within a model.Thetrain_model_iterativelyfunction iteratively trains a model utilizing early stopping and active sparsification on a per-task basis.NOTEthat this function only works on modelswithoutMultiMaskedConv2Dlayers.LayersThemann.layerssubpackage contains custom Keras-compatible layers which can be used to train sparse multitask models. The layers contained in this package are as follows:MaskedDenseThis layer is nearly identical to the Keras Dense layer, but it supports masking and pruning to reduce the number of active weights.MaskedConv2DThis layer is nearly identical to the Keras Conv2D layer, but it supports masking and pruning to reduce the number of active weights.MultiMaskedDenseThis layer supports isolating pathways within the network and dedicating them for individual tasks and performing fully-connected operations on the input data.MultiMaskedConv2DThis layer supports isolating pathways within the network and dedicating them for individual tasks and performing convolutional operations on the input data.MultiDenseThis layer supports multitask inference using a fully-connected architecture and is not designed for training. Once a model is trained with theMultiMaskedDenselayer, that layer can be converted into this layer for inference by using themann.utils.remove_layer_masksfunction.MultiConv2DThis layer supports multitask inference using a convolutional architecture and is not designed for training. Once a model is trained with theMultiMaskedConv2Dlayer, that layer can be converted to this layer for inference by using themann.utils.remove_layer_masksfunction.SelectorLayerThis layer selects which of the multiple inputs fed into it is returned as a result. This layer is designed to be used specifically with multitask layers.SumLayerThis layer returns the element-wise sum of all of the inputs.FilterLayerThis layer can be turned on or off, and indicates whether the single input passed to it should be output or if all zeros should be returned.MultiMaxPool2DThis layer implements Max Pool operations on multitask inputs.Additional Documentation and Training MaterialsAdditional documentation and training materials will be added to theBeyondML Documentation Websiteas we continue to develop this project and its capabilities.Feature RoadmapPyTorch SupportWe are currently working on building support for PyTorch models and layers into this packageFixing issues with iterative training andMultiMaskedConv2DlayersAs mentioned above, there are issues with finding pert-task gradients with models utilizingMultiMaskedConv2Dlayers.  Future iterations of the technology will address bugs with these kinds of modelsChangesBelow are a list of additional features, bug fixes, and other changes made for each version.Version 0.2.2Small documentation changesAddedquantize_modelfunctionAddedbuild_transformer_blockandbuild_token_position_embedding_blockfunctions for transformer functionalityRemoved unnecessary imports breaking imports in minimal environmentsVersion 0.2.3Per-task pruningFunctionality for this feature is implemented, but usage is expected to be incomplete. Note that task gradients have to be passed retrieved and passed to the function directly (helper function available), and that the model has to initially be compiled using a compatible loss function (recommended 'mse') to identify gradients.It has been found that this functionality is currently only supported for models with the following layers:MaskedConv2DMaskedDenseMultiMaskedDenseNote also that this functionality does not support cases where layers of an individual model are other TensorFlow models, but supporting this functionality is on the roadmap.Iterative training using per-task pruningFunctionality for this feature is implemented, but there are known bugs when trying to apply this methodology to models with theMultiMaskedConv2Dlayer presentVersion 0.3.0Support for PyTorch layersSupport for additional custom objects in thequantize_modelfunctionAdded tests to the package functionalityAdded auto-generated documentation