Authentication protocol. An authentication protocol is a type of computer communications protocol or cryptographic protocol specifically designed for transfer of authentication data between two entities. It allows the receiving entity to authenticate the connecting entity (e.g. Client connecting to a Server) as well as authenticate itself to the connecting entity (Server to a client) by declaring the type of information needed for authentication as well as syntax. It is the most important layer of protection needed for secure communication within computer networks.

Purpose 

With the increasing amount of trustworthy information being accessible over the network, the need for keeping unauthorized persons from access to this data emerged. Stealing someone's identity is easy in the computing world - special verification methods had to be invented to find out whether the person/computer requesting data is really who he says he is. The task of the authentication protocol is to specify the exact series of steps needed for execution of the authentication. It has to comply with the main protocol principles:
 A Protocol has to involve two or more parties and everyone involved in the protocol must know the protocol in advance.
 All the included parties have to follow the protocol.
 A protocol has to be unambiguous - each step must be defined precisely.
 A protocol must be complete - must include a specified action for every possible situation.
An illustration of password-based authentication using simple authentication protocol:

Alice (an entity wishing to be verified) and Bob (an entity verifying Alice's identity) are both aware of the protocol they agreed on using. Bob has Alice's password stored in a database for comparison. 
 Alice sends Bob her password in a packet complying with the protocol rules.
 Bob checks the received password against the one stored in his database. Then he sends a packet saying "Authentication successful" or "Authentication failed" based on the result.
This is an example of a very basic authentication protocol vulnerable to many threats such as eavesdropping, replay attack, man-in-the-middle attacks, dictionary attacks or brute-force attacks. Most authentication protocols are more complicated in order to be resilient against these attacks.

Types

Authentication protocols developed for PPP Point-to-Point Protocol
Protocols are used mainly by Point-to-Point Protocol (PPP) servers to validate the identity of remote clients before granting them access to server data. Most of them use a password as the cornerstone of the authentication. In most cases, the password has to be shared between the communicating entities in advance.

PAP - Password Authentication Protocol
Password Authentication Protocol is one of the oldest authentication protocols. Authentication is initialized by the client sending a packet with credentials (username and password) at the beginning of the connection, with the client repeating the authentication request until acknowledgement is received. It is highly insecure because credentials are sent "in the clear" and repeatedly, making it vulnerable even to the most simple attacks like eavesdropping and man-in-the-middle based attacks. Although widely supported, it is specified that if an implementation offers a stronger authentication method, that method must be offered before PAP. Mixed authentication (e.g. the same client alternately using both PAP and CHAP) is also not expected, as the CHAP authentication would be compromised by PAP sending the password in plain-text.

CHAP - Challenge-handshake authentication protocol

The authentication process in this protocol is always initiated by the server/host and can be performed anytime during the session, even repeatedly. The server sends a random string (usually 128B long). The client uses the password and the string received as input to a hash function and then sends the result together with username in plain text. The server uses the username to apply the same function and compares the calculated and received hash. An authentication is successful when the calculated and received hashes match.

EAP - Extensible Authentication Protocol

EAP was originally developed for PPP(Point-to-Point Protocol) but today is widely used in IEEE 802.3, IEEE 802.11(WiFi) or IEEE 802.16 as a part of IEEE 802.1x authentication framework. The latest version is standardized in RFC 5247. The advantage of EAP is that it is only a general authentication framework for client-server authentication - the specific way of authentication is defined in its many versions called EAP-methods. More than 40 EAP-methods exist, the most common are:
EAP-MD5
EAP-TLS
EAP-TTLS
EAP-FAST
EAP-PEAP

AAA architecture protocols (Authentication, Authorization, Accounting)

Complex protocols used in larger networks for verifying the user (Authentication), controlling access to server data (Authorization) and monitoring network resources and information needed for billing of services (Accounting).

TACACS, XTACACS and TACACS+
The oldest AAA protocol using IP based authentication without any encryption (usernames and passwords were transported as plain text). Later version XTACACS (Extended TACACS) added authorization and accounting. Both of these protocols were later replaced by TACACS+. TACACS+ separates the AAA components thus they can be segregated and handled on separate servers (It can even use another protocol for e.g. Authorization). It uses TCP (Transmission Control Protocol) for transport and encrypts the whole packet. TACACS+ is Cisco proprietary.

RADIUS

Remote Authentication Dial-In User Service (RADIUS) is a full AAA protocol commonly used by ISPs. Credentials are mostly username-password combination based, and it uses NAS and UDP protocol for transport.

DIAMETER
Diameter (protocol) evolved from RADIUS and involves many improvements such as usage of more reliable TCP or SCTP transport protocol and higher security thanks to TLS.

Other

Kerberos (protocol)
Kerberos is a centralized network authentication system developed at MIT and available as a free implementation from MIT but also in many commercial products. It is the default authentication method in Windows 2000 and later. The authentication process itself is much more complicated than in the previous protocols - Kerberos uses symmetric key cryptography, requires a trusted third party and can use public-key cryptography during certain phases of authentication if need be.

List of various other authentication protocols
 AKA
 Basic access authentication
 CAVE-based authentication
 CRAM-MD5
 Digest
 Host Identity Protocol (HIP)
 LAN Manager
 NTLM, also known as NT LAN Manager
 OpenID protocol
 Password-authenticated key agreement protocols
 Protocol for Carrying Authentication for Network Access (PANA)
 Secure Remote Password protocol (SRP)
 RFID-Authentication Protocols
 Woo Lam 92 (protocol)
 SAML

References

 
Computer access control protocols