Remove select from worker loop, refactor shared memory things, lots of fixes