Hot air balloons can only barely control their direction. Each type of aviation activity must be conducted within its design limits. Accident statistics are difficult to evaluate. Should it be expressed as accidents or fatalities. Do you want it per mile, per hour, per flight, or per pilot.
Airlines use seat-miles to get the best possible numbers.
All the various types of established recreational flying are reasonably safe if you follow good practices. First they are not marginal. Ultralights are designed to have the same structural strength as regular normal category aircraft. A major reason people fly them is the lower cost. In spite of what critics might say by comparing the cost of an old worn out conventional aircraft with a new ultralight, the average cost of owning and flying an ultralight is much less than conventional aircraft.