Byzantine failure occurs when a node behaves abnormally and sends different messages to different peers. This type of failure is challenging to handle because it can lead to inconsistencies within the system.