# Deep Residual Learning for Image Recognition: A Survey
## 4 Deep Residual Learning for Image Recognition
### Deep Residual Learning Hyperparameter Optimization  
Deep Residual Learning Hyperparameter Optimization [28] is a method for optimizing the hyperparameters of a Deep Residual Learning model. When optimizing the parameters of a deep residual network for image recognition, there are many factors to consider. The first is the depth of the network. Deeper networks have more layers and can therefore learn more complex features. Shallower networks, on the other hand, are faster to train and may be more efficient in terms of memory usage. Another important parameter is the width of the network, which refers to the number of neurons in each layer. Wider networks can learn more complex features, but are also more expensive to train. Finally, the learning rate is an important parameter that controls how quickly the network learns from data. A higher learning rate means that the network learns faster, but may also be more likely to overfit on the training data. If we use a standard gradient descent algorithm to optimize the parameters of a neural network such as this one, it is not guaranteed that we will find good local minima. It would be better if we had some way of knowing what global minima might look like before starting optimization so that we could search intelligently instead of randomly guessing where they might be.  
Recent work has attempted to do this by using linear classifiers such as SVM's or k-nearest neighbors to label different regions in feature space based on whether they corresponded with positive or negative examples respectively. Then, these labeled regions were used as starting points for gradient descent optimization, providing us with potential solutions near these regions instead of random ones. While this technique provides us with valuable insight into where to start optimization, it still suffers from problems because these labels aren't perfect. For example, the labels are only correct 50% of the time, meaning that our search process is less accurate than we would hope. More research is needed to improve upon this technique and make sure that it gives reliable results every time. However, despite its imperfections, this approach does provide new insights into the problem of hyperparameter optimization for deep networks and may help lead to improved methods in the future. One of the main problems for optimization algorithms is that the gradients become too small when you go down into lower layers. Therefore, all traditional optimizers try to converge at a single minimum when in reality there are multiple local minima. One solution proposed was using non-convex strategies like Bayesian Optimization (BO) and genetic algorithms (GA). In BO you estimate gradients at many different points simultaneously, while GA does not rely on gradient information at all. Since both techniques show promise, further research should explore their performance in practice.  
There are many factors to consider when optimizing the parameters of a deep residual network for image recognition. This paper discussed three key considerations: the depth of the network, width of the network, and learning rate. There is currently much ongoing research exploring novel ways to automate this process; however, no clear winner has emerged yet. There are many issues that need to be addressed, including gradient size and convexity. Other approaches include Bayesian optimization and genetic algorithms.