# Deep Residual Learning for Image Recognition: A Survey
## 8 Traditional Deep Learning Methods
### ResNet Architecture  
Deep residual learning is a neural network architecture that allows for very deep networks, typically tens or hundreds of layers, to be trained. This is in contrast to the traditional approach of training shallower networks with fewer layers. The benefits of deeper networks are well-known: they can learn features at increasingly higher levels of abstraction, which leads to better performance on a variety of tasks. However, training very deep networks has been difficult in the past due to the vanishing gradient problem, where the gradients of the error signal tend to become very small as they are propagated back through the network. The ResNet architecture was designed to address this issue by using skip connections between layers which allow the gradients to flow more easily through the network. Recently, ResNets have found success in tackling some of the most challenging problems in computer vision, such as image recognition. Similarly, deep residual learning for image recognition (DRL) has emerged as a powerful tool that helps to tackle many practical challenges and opens up exciting new avenues for future research. DRL for image recognition, similarly, provides two major advantages over other approaches: DRL does not require separate preprocessing steps before the CNN starts learning, and it performs surprisingly well without any architectural changes. Although it was inspired by early attempts at training very deep networks in computer vision, such as CaffeNet [87], DRL has since found success in tackling many practical challenges and opened up exciting new avenues for future research. Perhaps most significantly, a simple extension of the standard DRL approach (called deep residual learning) has successfully been applied to AlexNet [88], GoogLeNet [88], VGGNet [89], ResNeXt [90], Xception [46], as well as other popular image recognition models. This idea has now been used enough times to warrant a review; there are now two entire papers dedicated just to reviewing its history. The original application of DRL in the context of deep convolutional neural networks dates back to [PERSON] et al. 's 1998 paper Gradient-based learning that was applied to document recognition. Here, [PERSON] et al. proposed applying ReLU activation functions combined with max pooling within each layer rather than across layers. This would lead to shallow but wide convolutional neural networks capable of compressing multiple characters into one neuron for every letter inputted into the system. At that time, according to [PERSON] et al., state-of-the art systems were about 25% accurate. Similar results were obtained when [PERSON] et al. (author in studied) performed a similar experiment with AlexNet, achieving 94% accuracy with the same set of parameters. With the help of DRL, this network was able to exceed 98% accuracy on ImageNet, which was previously achieved by a variant of Wide Residual Networks (WRNs). This result is particularly noteworthy because WRNs are often thought to be superior to DRL when it comes to neural networks with large fully-connected layers. Some other examples of successful implementations of DRL are shown below:  
Similarly, deep residual learning for image recognition has recently been applied to building Multimodal Retrieval Models. Deep residual learning for image recognition has recently been applied to building Multimodal Retrieval Models. This novel technique involves a model that utilizes both pixel-wise representations of images and corresponding captions. These captions may be obtained in either a one-shot fashion or iteratively, in a more laborious manner, with the ultimate goal of reducing the gap between manual annotations provided by human experts and automated annotations. For example, take the case of captioned photos. A common technique for creating these photographs involves having somebody write the description they imagine will accompany the photo while the photo is being taken (sometimes called instructing what to say).