# Deep Residual Learning for Image Recognition: A Survey
## 5 Image Recognition
### Image Recognition Deep Learning  
In recent years, deep learning methods have revolutionized the field of image recognition. The author in [67] studied deep residual learning (ResNet) and found that it can outperform previous state-of-the-art methods by a large margin. In order to address over-fitting in deep learning algorithms, regularization is often used, though there are many different types of regularization strategies and they need careful tuning. One popular strategy is weight decay (WD) [77], which penalizes large weights within an activation function. Larger penalties lead to more reasonable parameter choices, leading to better generalization. Other common strategies include dropout (which randomly sets some nodes to zero in the network during training) and l2 regularization. The two main strategies for training deep residual nets are online gradient descent with momentum and batch gradient descent. They found that online gradient descent was better suited for small batches, while batch gradient descent performed better when large batches were used. Interestingly, their experiments showed that neither of these two strategies performed significantly better than the other on the ImageNet dataset. Another important aspect of training is data augmentation. Augmenting the images with various transformations improves performance dramatically because the neural network learns representations invariant to translation, scaling, rotation, and brightness changes. Finally, let's talk about some real world applications for deep residual nets. Computer vision has seen significant advances due to improvements in hardware, algorithm development and new datasets. Some great real world use cases include video processing where these models can be applied at multiple stages from raw frames all the way up to final video output; pedestrian detection [78], where we want systems that detect people who walk across camera views; and traffic sign detection, where there are thousands of variations of signs that must be identified correctly. These deep residual networks represent powerful tools in the arsenal of machine learning practitioners. To conclude, I would like to mention the following five limitations of deep residual nets. First, there is no principled way to choose the depth of a network, meaning that researchers must rely on experimentation to determine the optimal depth. Second, even though data augmentation helps reduce overfitting, it cannot eliminate it entirely. Third, if dealing with objects that are highly non-rigid (e.g., hair), then fine tuning per frame becomes computationally prohibitive and requires other machine learning techniques instead. Fourth, it is hard to train the last layer with backpropagation, especially for deep networks. Fifth, in practice, the first few layers of a deep network usually work better and are easier to optimize. One limitation is that there is no principled way to decide on the optimal depth of a network, and this needs to be determined through experimentation. Data augmentation can help reduce overfitting but it can't eliminate it entirely. Fine-tuning per frame becomes computationally prohibitive for non-rigid objects (e.g., hair), so other machine learning techniques are needed instead. It is hard to train the last layer with backpropagation, and in practice, the first few layers typically work better and are easier to optimize. One of the benefits of deep residual learning is that it reduces the dependence on manually chosen parameters, since most of them are learned by the network. There are several examples of successful deep residual net architectures for image recognition and computer vision. Some potential drawbacks of deep residual learning include vulnerability to adversarial examples and computational cost. There is a need for careful consideration before applying deep residual learning methods to any particular application area. Deep residual learning presents a powerful set of methods for solving problems in image recognition and computer vision.