We take C code and target that to the FPGA using the Celoxica software. The driver marshals and serialises data passed by the task, making a function call to a hardware-based routine. The multiplexer takes that data and passes it to the logic that implements the function. When the return data is ready, it is passed back through the multiplexer to the driver and on up to the calling task. To ensure realtime behaviour, the OS priority-queues requests for FPGA processing.
Tasks can make blocking or non-blocking calls.
In the latter case, the OS uses a call-back function to tell the task that the data is ready. People are out there looking at this from a sequential point of view. The key to the architecture is determining where your bottlenecks lie. We have taken a triple-DES encryption algorithm. That's when you start finding other bottlenecks, such as the speed of the hard disk.