Package: gil-galad
A deep learning project template for TensorFlow. Utilities for TFDataset handling and hyperparameter optimization.Getting StartedFork the repository, navigate to the local directory and follow the quick start checklist below. Data should be divided into training, validation and testing sets and placed in .tfrecords file format in separate directories. Utilities for converting NumPy arrays to .tfrecords files are included in the utils.py module.Quick Start ChecklistBuild neural network models in models.pyNote:hyperparameters that will be optimized later need to be specified. SeeModel Selectionsection below.Connect models, define loss, evaluation metrics, optimizer, etc. in graph.pyNote:hyperparameters that will be optimized later need to be specified. SeeModel Selectionsection below.Specify input data shapes in data_shapes dictionaryNote:Data shapes should not include batch sizeâ€”this information is passed to the sampler insteadAdditional note:the order in data_shapes should correspond to the order you retrieve the tensors in the graphInmain.py:data_shapes = {'lowres': (32, 32, 3), 'highres': (128, 128, 3)}Ingraph.py:lowres, highres = data.get_batch()Create a DataSampler object with the filepaths to your train/valid/test sets and the data shapes dictionary.Define hyperparameters to optimize over and their corresponding domain ranges in dictionaryPass your graph object and hyperparameter dictionary to Sherpa viagilgalad.opt.bayesian_optimization(Optional) Add custom plotting functionality in plotting.py. Send any matplotlib figure to TensorBoard via tfplot.Model SelectionGil-Galad model selection is employed via Sherpa's Bayesian optimization suite which utilizes sklearn's Gaussian process module. Bayesian optimization specifies a distribution over functions via a kernel function and prior. Here, the mean function corresponds to a surrogate objective function whose predictor variables are the model hyperparameters. The prior distribution over functions is updated via Bayes' rule to account for trial runs wherein the independent variables specify the model and the dependent variable is the evaluation of such a model on the validation dataset.With Gil-Galad, we specify which hyperparameters we will optimize by passing a parameter dictionary to our graph class while also defining default hyperparameters during graph and model construction as follows:Graph-levelclassGraph(BaseGraph):def__init__(self,network,sampler,logdir=None,ckptdir=None):self.network=networkself.data=samplerself.logdir=logdirself.ckptdir=ckptdirself.build_graph()defbuild_graph(self,params=None):tf.reset_default_graph()self.data.initialize()self.x,self.y,self.z=self.data.get_batch()self.y_=self.network(self.x,params=params)self.loss=tf.losses.mean_squared_error(self.y,self.y_)update_ops=tf.get_collection(tf.GraphKeys.UPDATE_OPS)withtf.control_dependencies(update_ops):opt=tf.train.AdamOptimizer(learning_rate=params['lr']ifparamselse0.001)self.update=opt.minimize(loss=self.loss,var_list=self.network.vars,global_step=self.global_step)Model-levelclassModel(BaseModel):def__init__(self,name):self.name=namedef__call__(self,x,params):withtf.variable_scope(self.name)asvs:y=conv_2d(x=x,filters=params['filters']ifparamselse64,kernel_size=params['kernel_size']ifparamselse3,strides=2,activation=params['activation']ifparamselse'relu')returnyWe then define the hyperparameter domain type and ranges in a dictionary. This information accompanies the graph object as arguments for the Bayesian optimization function.importgilgaladasgghyperparameters={'Discrete':{'filters':[64,128],'kernel_size':[3,5]},'Continuous':{'lr':[1e-5,1e-3]},'Choice':{'activation':['relu','prelu']}}best_model=gg.opt.bayesian_optimization(graph=graph,params=hyperparameters,max_trials=50)