Fix matrix shape when swapping bytes