## 2 History of ANN Development  
ANNs provide a method to characterize synthetic neurons to solve complex problems in the same manner as the human brain. For many years, especially since the middle of the last century, an interest in studying the brain's mechanism and structure has been increasing. This growing research interest has led to the development of new computational models, connectionist systems or ANNs, based on the biological background, for solving complex problems like pattern recognition, and fast information processing and adaptation.  
In the early 1940s, [PERSON] and [PERSON] [37] studied the potential and capabilities of the interconnection of several basic components, based on the model of a neuron. Later on, others, like [PERSON] [38], were concerned with the adaptation laws involved in neural systems. In 1958, [PERSON] [39] coined the name Perceptron and devised an architecture which has subsequently received much attention. In 1960, [PERSON] and his student, [PERSON], [40] presented an important generalization of the perceptron training algorithm as the Least Mean Square (LMS) learning procedure, also known as the Delta rule. The learning rule was applied as the adaptive linear element in the ADALINE (ADAptive LInear Neuron) networks. Then, [PERSON] and [PERSON] [41] introduced a rigorous analysis of the Perceptron, of which they proved many properties and pointed out limitations of several related models. In the 1970s, the work of [PERSON] [42] came to prominence. His work, based on biological and psychological evidence, proposed several novel architectures of nonlinear dynamic systems. In 1982, [PERSON] [43] applied a particular nonlinear dynamic structure to solve problems in optimization. All of them conducted pioneer studies on the theoretical aspect of ANNs, particularly starting in the 1950s and 1960s.  
In 1986, the PDP research group published a series of algorithms and results [2]. This publication contains an article entitled \"Learning Internal Representations by Error Propagation\" [1]. This article made the recognition of the BP training algorithm although it was already described in 1974 [44]. The BP algorithm implemented with the general Delta rule [1, 7], as the representative of supervised training algorithms, gave a strong impulse to the subsequent research and has resulted in the largest body of research and applications in ANNs although a number of other ANN architectures and training algorithms have been developed and applied at the same time. In 1982, Finnish Professor [PERSON] [45] first described a different ANN architecture which is trained using an unsupervised SOM learning procedure. This Kohonen SOM algorithm has been populated in many research and practical applications later on. In 1988, RBF neural networks were first used [46], although RBFs were introduced earlier in 1977 [47]. RBF networks have also been widely used with the strong capability of function approximation and, along with MLP perceptron, had the impact on the emergence of SVMs [32, 33, 48]. In the late 1980s, the standard MFNs were proven as universal approximators on a compact subset of R\({}^{\text{n}}\)[49, 50]. The MFNs have a single hidden layer containing finite number of hidden neurons. The neurons use arbitrary activation function The theory assures that MFNs, including MLP networks, RBF networks and even SVMs under certain conditions [51] can handle problems which are highly complex and nonlinear.  
So far, ANNs have been used in many industrial and commercial applications such as process modeling and control [52], character recognition [53], image recognition [54], credit evaluation [55], fraud detection [56, 57], insurance [58], and stock forecasting [59]. In a later section, applications of ANNs in agricultural and biological engineering will be reviewed.