Actually keep the delay/threshold in the listener struct.

This makes it consistent across all incoming connections, for real this
time (oops).