From what I can tell between my own experiments and postings in cainn it may be that the very slow online update method will sometimes give the best results so what you get from these acceleration algorithms may be slightly less than what you can achieve with this pitifully slow method. Having said that, in most cases the acceleration algorithms work so much faster than either online or batch backprop you should try them first and then if you want slightly better results you can try the slower online update method. Given that online updating is so slow you may give up on it long before you get a network trained. As the training proceeds you increase eta in some way if you keep going downhill in terms of the error. When the weight change gets too large you end up on the other side of the valley and for this you must decrease the learning rate in some way.
Algorithms vary on both the speeding up and slowing down details.
Second there is a set of algorithms known as conjugate gradient methods. Third there are the methods that build up the network one hidden unit at a time. While rprop may be the best choice in more cases than any other it is not the best in every case, from time to time I run into a problem where rprop behaves poorly. Quickprop Quickprop is also one of the better training algorithms and is loosely based on Newton's method for finding the root of a quadratic function. The article on this by Scott Fahlman is available online .