Well that was subtle.

Turns out that most of the time the kernel just assumes that if we have a physical NIC with a TOE on it then it can be assumed to take care of all duties of calculating correct checksums in IP/TCP/UDP. As a result, when you initially send a packet, the checksums will be wrong, until they leave the machine.

Kernels will drop (sometimes silently!) packets they receive which have got the wrong checksums.

Because the router was picking up packets, wrapping them, and then sending then to a remote router, when they get unwrapped there, they were having the wrong checksums, and so the kernel then drops them.

The solution is to ensure that when each contain boots, it turns off all assumptions about the veth devices its been given and their capacity for TOE. Consequently, we now get the correct checksums inserted as soon as the packet enters the SDN, and they stay correct as they are wrapped, moved across to a remote router and unwrapped. Connectivity appears stable now.