Minor refactoring

While not quite as conceptually nice as before,  this is shorter and
eliminates an readIORef and withForeignPtr in some functions