The traditional computers that we deal with every day have changed very little since their beginnings in the 1940's. The usefulness of the computer lies in its vast speed at executing those rules - it is a superb machine but not a mind. Neural networks are very different - they are composed of many rather feeble processing units which are connected into a network. Their computational power depends on working together on any task - this is sometimes termed parallel processing . There is no central CPU following a logical sequence of rules - indeed there is no set of rules or program.
Computation is related to a dynamic process of node firings.
This structure then is much closer to the physical workings of the brain and leads to a new type of computer that is rather good at a range of complex tasks.