If you don't file tax returns, the IRS can take as long as it wants to review your taxes and finalize your liability. If the IRS determines that you owe tax, then the statute of limitations starts at that point. If it audits your taxes and determines that you were actually owed a refund after the three-year statute of limitations has passed, then you lose that refund.