You should always drive slowly with a paddle board on your car. The faster you go, the more likely it is that your board will come loose. Some hard roof rack manufacturers state a maximum speed of 55mph that you shouldn’t exceed.
It’s a good idea to drive more slowly if you are using a soft rack or homemade system. Always stop driving about five minutes after you’ve set off to check that the board hasn’t moved.