A broken chair is not a chair
A broken chord is not a chord.
An inverted pendulum is not a pendulum. Never leave out the the "inverted" in "inverted pendulum".

The problem with small wheeels and no castor is that the rider had to ride a combined inverted pendulum with gyroscopic forces that inhibit steering,  and then feed back into "speed wobble" ..instability at speed.

So riding slowly, one can ride the inverted pendulum .   When riding the penny farthing, one had to slow down  to inverted pendulum speeds to turn, and then one had to be a master at inverted pendulum riding to achieve the turn.

Attempting to overcome gyrscopic force of the penny farthing front wheel while moving at speed was not possible, as the small and lightly loaded rear wheel would not grip and so would not provide enough torque. The small wheel would bounce along and provide very little torque.

The penny farthing got the large front wheel because the older castor deficient design did not have signficant castor, and were hard to pedal as the pedal 'plane' rotated compared to the rider (a real pain  in the legs !) 

The safety bike gave the much more leg friendly pedal arrangement , plus the castor on the front  wheel removed the problems with turning ,  stability at speed and feed back to the rider. The better pedal arrangement of the safety bike gave the rider better  ability to do the "inverted pendulum" control at slow speeds.