Init the datapattern buffer after worker init

This move is neccessary because soon the buffer won't be valid inside
of worker init. This change will also cause the datapattern buffers to
be initialized in parallel (whereas before it was serialized inside of
worker init).
Allocate source buffers after worker init
Factor out common e2e connect code