Rounding up

Rounding-up is a phenomenon that occurs in sailing when the helmsman (or tiller-handler) is no longer able to control the direction of the boat and it heads up (or "rounds up") into the wind, causing the boat to slow down, stall out, or tack. This occurs when the wind overpowers the ability of the rudder to maintain a straight course.

For example, the sailboat may heel over so far that the rudder no longer engages in the water, or only to such a small extent that it can no longer steer the boat.  When this happens is dependent on a number of factors such as the velocity of the wind, design of the hull and rudder and shape of the sails. This can be dangerous if it causes collisions between boats if sailing close together. Rounding up can be startling to those on board: The boat turns into the eye of the wind with all sails fluttering. However, unless the boat is in irons (stopped facing directly into the wind), control can be regained by steering the boat off the wind again to refill the sails and regain the desired course.  Often the crew will need to ease out the sheets (lines that control the trim of the sail) before this can occur, in order to reduce the wind force on the sails.  Rounding up is in fact a safety design of most sailboats that can help prevent a knock-down and allow the helmsman to regain control of the boat.  An occasional round-up may simply be the result of a strong gust of wind.  If it occurs regularly, this may be a sign that too much sail is raised and the crew may need to lower one or more sails, change to smaller sails, or reef.  In smaller sailboats such as racing dinghies where the sail cannot be lessened or reefed, frequent round-ups may be a sign that the wind conditions are too strong for the boats and they should immediately proceed to shelter.

Sail settings to reduce rounding up
The best way to reduce rounding up in an over-canvassed boat, is to reduce sail by reefing, lowering or furling. If this is not possible due to rig design, for example, then sail adjustment may reduce or prevent rounding up when beating to windward. For a fore-and-aft rigged craft:
 The vang may be loosened, the mainsheet traveller moved to windward, or both, to allow the boom rise
 The mainsheet may be loosened to spill wind
Allowing the boom to rise increases the twist in the mainsail, which spills wind higher in the sail while keeping power in the bottom of the mainsail to maintain the effectiveness of the rudder.  
If a boat tends to round up when the helmsman is attempting to sail straight downwind ("running down before the wind"), and you cannot reduce sail area, but the boat is manageable on the other points of sail, then the only solution is to avoid running down before the wind by instead tacking downwind.   This means you do not head straight for a downwind destination, but rather follow a zig-zag course, arriving at your downwind destination through a series of reaches.   Letting out the mainsail more by loosening the sheet, or allowing the boom to rise by loosening the boom vang or the traveler, will not help when running down before the wind.   Also futile for this problem are adjustments aiming to move the center of effort relative to the center of lateral resistance—these only help for points of sail other than running down before.

References

Sailing