It's all about what counts as a non-taxpayer, and it does get complicated as there's something called the tax-free savings allowance for lower incomes, which is a separate £5,000 tax-free allowance for people who just have income from savings.
The crucial part is that even if you have savings interest, as long as you're not a taxpayer, you can still apply for and benefit from the marriage tax allowance.