IPsec operates in two modes: Transport mode and Tunnel mode. Transport mode encrypts the data payload of the IP packet, while Tunnel mode encrypts both the IP packet header and payload. Tunnel mode is often used for virtual private networks (VPNs) to secure data transmission between networks.