This attack needs a malicious (or compromised) hardware wallet manufacturer. If someone breaks into your computer, there is no way for them to exploit this attack vector.