Package: gradient-centralization-tf
Gradient Centralization TensorFlowThis Python package implements Gradient Centralization in TensorFlow, a simple and effective optimization technique for
Deep Neural Networks as suggested by Yong et al. in the paperGradient Centralization: A New Optimization Technique for Deep Neural Networks. It can both speedup training
process and improve the final generalization performance of DNNs.InstallationRun the following to install:pipinstallgradient-centralization-tfAbout the Examplesgctf_mnist.ipynbThis notebook shows the the process of using thegradient-centralization-tfPython package to train on theFashion MNISTdataset availaible fromtf.keras.datasets. It further
also compares usinggctfand performance without usinggctf.gctf_horses_v_humans.ipynbThis notebook shows the the process of using thegradient-centralization-tfPython package to train on theHorses vs Humansdataset byLaurence Moroney. It further also compares usinggctfand performance without usinggctf.Usagegctf.centralized_gradients_for_optimizerCreate a centralized gradients functions for a specified optimizer.Arguments:optimizer: atf.keras.optimizers.Optimizer object. The optimizer you are using.Example:>>>opt=tf.keras.optimizers.Adam(learning_rate=0.1)>>>opt.get_gradients=gctf.centralized_gradients_for_optimizer(opt)>>>model.compile(optimizer=opt,...)gctf.get_centralized_gradientsComputes the centralized gradients.This function is ideally not meant to be used directly unless you are building a custom optimizer, in which case you
could pointget_gradientsto this function. This is a modified version oftf.keras.optimizers.Optimizer.get_gradients.Arguments:optimizer: atf.keras.optimizers.Optimizerobject. The optimizer you are using.loss: Scalar tensor to minimize.params: List of variables.Returns:A gradients tensor.gctf.optimizersPre built updated optimizers implementing GC.This module is speciially built for testing out GC and in most cases you would be usinggctf.centralized_gradients_for_optimizerthough this module implementsgctf.centralized_gradients_for_optimizer. You can directly use all optimizers withtf.keras.optimizersupdated for GC.Example:>>>model.compile(optimizer=gctf.optimizers.adam(learning_rate=0.01),...)>>>model.compile(optimizer=gctf.optimizers.rmsprop(learning_rate=0.01,rho=0.91),...)>>>model.compile(optimizer=gctf.optimizers.sgd(),...)Returns:Atf.keras.optimizers.Optimizerobject.DevelopinggctfTo installgradient-centralization-tf, along with tools you need to develop and test, run the following in your
virtualenv:gitclonegit@github.com:Rishit-dagli/Gradient-Centralization-TensorFlow# or clone your own forkpipinstall-e.[dev]LicenseCopyright 2020 Rishit Dagli

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.