Path MTU Discovery

Path MTU Discovery (PMTUD) is a standardized technique in computer networking for determining the maximum transmission unit (MTU) size on the network path between two Internet Protocol (IP) hosts, usually with the goal of avoiding IP fragmentation. PMTUD was originally intended for routers in Internet Protocol Version 4 (IPv4). However, all modern operating systems use it on endpoints.  In IPv6, this function has been explicitly delegated to the end points of a communications session.
As an extension to the standard path MTU discovery, a technique called Packetization Layer Path MTU Discovery works without support from ICMP.

Implementation
For IPv4 packets, Path MTU Discovery works by setting the Don't Fragment (DF) flag bit in the IP headers of outgoing packets. Then, any device along the path whose MTU is smaller than the packet will drop it, and send back an Internet Control Message Protocol (ICMP) Fragmentation Needed (Type 3, Code 4) message containing its MTU, allowing the source host to reduce its path MTU appropriately. The process is repeated until the MTU is small enough to traverse the entire path without fragmentation.

As IPv6 routers do not fragment packets, there is no Don't Fragment option in the IPv6 header. For IPv6, Path MTU Discovery works by initially assuming the path MTU is the same as the MTU on the link layer interface where the traffic originates.  Then, similar to IPv4, any device along the path whose MTU is smaller than the packet will drop the packet and send back an ICMPv6 Packet Too Big (Type 2) message containing its MTU, allowing the source host to reduce its path MTU appropriately. The process is repeated until the MTU is small enough to traverse the entire path without fragmentation.

If the path MTU changes after the connection is set up and becomes lower than the previously determined path MTU, the first large packet will cause an ICMP error and the new, lower path MTU will be found. If the path changes and the new path MTU is larger, the source will not learn about the increase, because all routers along the new path will be capable of relaying all packets that the source sends using the originally determined, lower path MTU.

Problems
Many network security devices block all ICMP messages for perceived security benefits, including the errors that are necessary for the proper operation of PMTUD.  This can result in connections that complete the TCP three-way handshake correctly, but then hang when data are transferred.  This state is referred to as a black hole connection.

Some implementations of PMTUD attempt to prevent this problem by inferring that large payload packets have been dropped due to MTU rather than because of link congestion.  However, in order for the Transmission Control Protocol (TCP) to operate most efficiently, ICMP Unreachable messages (type 3) should be permitted.  A robust method for PMTUD that relies on TCP or another protocol to probe the path with progressively larger packets has been standardized in RFC 4821.

A workaround used by some routers is to change the maximum segment size (MSS) of all TCP connections passing through links with MTU lower than the Ethernet default of 1500. This is known as MSS clamping.

References

Computer network analysis
Internet protocols