Add panic for invalid pool sizes

Since this is perfectly avoidable by the user, a raw panic is
acceptable.