Not sending unnecessary packets *significantly* helped the delay, which does seem to be caused by a backup of packets in the queue