The main drawback is that secure chips are closed source. Firmware on a secure chip can't be open-sourced due to non-disclosure agreements, making independent audits challenging.