So, your rates of oxygen consumption were likely faster than you actually measured. A percentage of even the best mitochondria preparations have damaged mitochondria that cannot maintain respiratory control. They are the primary reason for a measurable state IV rate. The greater the percentage of damaged organelles, the greater the state IV rate. This means that more of the energy is lost as heat and is unavailable for transfer to ADP and inorganic phosphate.
Some energy is lost due to symport and other normal processes.
This holds for isolated mitochondria as well. In living cells mitochondria are never idle. They do exercise respiratory control, with electron transport slowing or increasing as the energy in the chemiosmotic gradient is decreasingly or increasingly utilized. Your demand for oxygen, reflected in the rate at which you breathe, is directly related to the rate of electron transport by mitochondria.