Assign thread affinity correctly