What limits the sensitivity of high altitude observatories