Package: deepswarm
Neural Architecture Search Powered by Swarm Intelligence 🐜DeepSwarmDeepSwarm is an open-source library which uses Ant Colony Optimization to tackle the neural architecture search problem. The main goal of DeepSwarm is to automate one of the most tedious and daunting tasks, so people can spend more of their time on more important and interesting things. DeepSwarm offers a powerful configuration system which allows you to fine-tune the search space to your needs.Example 🖼fromdeepswarm.backendsimportDataset,TFKerasBackendfromdeepswarm.deepswarmimportDeepSwarmdataset=Dataset(training_examples=x_train,training_labels=y_train,testing_examples=x_test,testing_labels=y_test)backend=TFKerasBackend(dataset=dataset)deepswarm=DeepSwarm(backend=backend)topology=deepswarm.find_topology()trained_topology=deepswarm.train_topology(topology,50)Installation 💾Install the packagepipinstalldeepswarmInstall one of the implemented backends that you want to usepipinstalltensorflow-gpu==1.13.1Usage 🕹Create a new file containing the example codetouchtrain.pyCreate settings directory which containsdefault.yamlfile. Alternatively you can run the script and instantly stop it, as this should automatically create settings directory which containsdefault.yamlfileUpdate the newly created YAML file to your dataset needs. The only two important changes you must make are: (1) change the loss function to reflect your task (2) change the shape of input and output nodesSearch 🔎(1) The ant is placed on the input node. (2) The ant checks what transitions are available. (3) The ant uses the ACS selection rule to choose the next node. (4) After choosing the next node the ant selects node’s attributes. (5) After all ants finished their tour the pheromone is updated. (6) The maximum allowed depth is increased and the new ant population is generated.Note: Arrow thickness indicates the pheromone amount, meaning that thicker arrows have more pheromone.Configuration 🛠Node typeAttributesInputshape: tuple which defines the input shape, depending on the backend could be (width, height, channels) or (channels, width, height).Conv2Dfilter_count: defines how many filters can be used.kernel_size: defines what size kernels can be used. For example, if it is set to [1, 3], then only 1x1 and 3x3 kernels will be used.activation: defines what activation functions can be used. Allowed values are: ReLU, ELU, LeakyReLU, Sigmoid and Softmax.Dropoutrate: defines the allowed dropout rates. For example, if it is set to [0.1, 0.3], then either 10% or 30% of input units will be dropped.BatchNormalization-Pool2Dpool_type: defines the types of allowed pooling nodes. Allowed values are: max (max pooling) and average (average pooling).pool_size: defines the allowed pooling window sizes. For example, if it is set to [2], then only 2x2 pooling windows will be used.stride: defines the allowed stride sizes.Flatten-Denseoutput_size: defines the allowed output space dimensionality.activation: defines what activation functions can be used. Allowed values are: ReLU, ELU, LeakyReLU, Sigmoid and Softmax.Outputoutput_size: defines the output size (how many different classes to classify).activation: defines what activation functions can be used. Allowed value are ReLU, ELU, LeakyReLU, Sigmoid and Softmax.SettingDescriptionsave_folderSpecifies the name of the folder which should be used to load the backup. If not specified the search will start from zero.metricsSpecifies what metrics should algorithm use to evaluate the models. Currently available options are: accuracy and loss.max_depthSpecifies the maximum allowed network depth (how deeply the graph can be expanded). The search is performed until the maximum depth is reached. However, it does not mean that the depth of the best architecture will be equal to the max_depth.reuse_patienceSpecifies the maximum number of times that weights can be reused without improving the cost. For example, if it is set to 1 it means that when some model X reuses weights from model Y and model X cost did not improve compared to model Y, next time instead of reusing model Y weights, new random weights will be generated.startSpecifies the starting pheromone value for all the new connections.decaySpecifies the local pheromone decay rate in percentage. For example, if it is set to 0.1 it means that during the local pheromone update the pheromone value will be decreased by 10%.evaporationSpecifies the global pheromone evaporation rate in percentage. For example, if it is set to 0.1 it means that during the global pheromone update the pheromone value will be decreased by 10%.greedinessSpecifies how greedy should ants be during the edge selection (the number is given in percentage). For example, 0.5 means that 50% of the time when ant selects a new edge it should select the one with the highest associated probability.ant_countSpecifies how many ants should be generated during each generation (time before the depth is increased).epochsSpecifies for how many epochs each candidate architecture should be trained.batch_sizeSpecifies the batch size (number of samples used to calculate a single gradient step) used during the training process.patienceSpecifies the early stopping number used during the training (after how many epochs when the cost is not improving the training process should be stopped).lossSpecifies what loss function should be used during the training. Currently available options are sparse_categorical_crossentropy and categorical_crossentropy.spatial_nodesSpecifies which nodes are placed before the flattening node. Values in this array must correspond to node names.flat_nodesSpecifies which nodes are placed after the flattening node (array should also include the flattening node). Values in this array must correspond to node names.verboseSpecifies if the associated component should log the output.Future goals 🌟Add a node which can combine the input from the two previous nodes.Add a node which can skip the depth n in order to connect to the node in depth n+1.Delete the models which are not referenced anymore.Add an option to assemble the best n models into one model.Add functionality to reuse the weights from the non-continues blocks, i.e. take the best weights for depth n-1 from one model and then take the best weights for depth n+1 from another model.Citation 🖋Online version is available at:arXiv:1905.07350@article{byla2019deepswarm,title={DeepSwarm: Optimising Convolutional Neural Networks using Swarm Intelligence},author={Edvinas Byla and Wei Pang},journal={arXiv preprint arXiv:1905.07350},year={2019}}Acknowledgments 🎓DeepSwarm was developed under the supervision ofDr Wei Pangin partial fulfilment of the requirements for the degree of Bachelor of Science of theUniversity of Aberdeen.