# A Review of the Optimal Design of Neural Networks Based on FPGA
## 2 The Development of Neural Networks  
As shown in Table 1, the development process of neural networks can be roughly divided into five stages: the proposal of the model, the stagnation period, the rise of the back propagation algorithm, the confusion period, and the rise of deep learning. The period of 1943-1958 is considered as the model proposal phase, and the perceptron wave proposed in 1958 continued for about 10 years. With increasingly more scholars entering into this direction of research, some scholars gradually found the limitations of perceptron model. With the publication of the Perceptron by [PERSON] in 1969, the neural network was pushed to the bottom directly, leading to the \"stagnation period\" of neural networks from 1969 to 1980. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Stage** & **Year** & **Character** & **Content** \\ \hline \multirow{6}{*}{Generation of models} & 1943 & [PERSON] and [PERSON] & McCulloch–Pitts model [15] \\ \cline{2-4}  & 1948 & [PERSON] & B-type Turing machine [16] \\ \cline{2-4}  & 1949 & [PERSON] & Hebb algorithm [17] \\ \cline{2-4}  & 1951 & [PERSON] and [PERSON] & The first neural network machine \\ \cline{2-4}  & 1958 & [PERSON] & Perceptron model [18] \\ \hline \multirow{6}{*}{Lag phase} & 1969 & [PERSON] & Perception [19] \\ \cline{2-4}  & 1974 & [PERSON] & Backpropogation (BP) algorithm [20] \\ \cline{2-4}  & 1980 & [PERSON] & Neocognitron model [21] \\ \hline \multirow{6}{*}{The rise of backpropagation algorithms} & 1982 & [PERSON] & Hopfield model [22] \\ \cline{2-4}  & 1985 & [PERSON] and Sejnowski & Boltzmann machine [23] \\ \cline{2-4}  & 1986 & [PERSON] and [PERSON] & Redescription of the BP algorithm [24] \\ \cline{2-4}  & 1989 & [PERSON] & Introducing the BP algorithm to the convolutional neural network [25] \\ \hline \multirow{2}{*}{Confusion period} & \multirow{2}{*}{1990–2005} & The rise of machine learning models has brought about great challenges to the development of neural networks \\ \hline \multirow{6}{*}{The rise of deep learning} & 2006 & [PERSON] & Deep Belief Networks [26] \\ \cline{2-4}  & 2012 & [PERSON] & AlexNet [27] \\ \cline{2-4}  & \multirow{6}{*}{2014} & [PERSON] & GoogLeNet [28] \\ \cline{2-4}  & & Visual Geometry Group and Google & VGGNet [29] \\ \cline{2-4}  & & DeepMind & \\ \cline{2-4}  & [PERSON] & GAN [30] \\ \cline{2-4}  & & [PERSON] and [PERSON] & DeepID [31] \\ \cline{2-4}  & & [PERSON] and [PERSON] & Region-CNN (RCNN) [32] \\ \cline{2-4}  & 2015 & [PERSON] & You Only Look Once (YOLOv1) [33] \\ \hline \multirow{6}{*}{2016} & \multirow{2}{*}{AlphaGo, an artificial intelligence machine developed by Google’s DeepMind, beat Go world champion [PERSON] 4-1} \\ \cline{2-4}  & & [PERSON] & YOLOv2 [34] \\ \cline{2-4}  & 2018 & [PERSON] & YOLOv3 [35] \\ \cline{2-4}  & 2020 & [PERSON] & YOLOv4 [36] \\ \cline{2-4}  & 2020–2022 & YOLOv5, YOLOv6 [37], YOLOv7 [38], etc. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Development history of neural networks. After 1980, increasingly more scholars paid attention to the back propagation algorithm, which reopened the thinking for researchers and launched another spring for the development of neural networks. Then, for a long period of time, scholars did not make breakthrough results, only working on the basis of existing research. In the mid-1990s, statistical learning theory and the machine learning model represented by the support vector machine began to rise. In contrast, the theoretical basis of neural networks was not clear, and optimization difficulties, poor interpretability, and other shortcomings became more prominent; thus, neural network research fell into a low tide again. Until 2006, Professor [PERSON], a neural network expert at the University of Toronto, and his students formally proposed the concept of deep learning. They proposed the deep belief network model in their published paper. [PERSON] et al. [26] found that the multi-layer feedforward neural network could be pre-trained layer by layer.