The later you file, the more difficult it is to accurately report income and deductions. And, it is likely the IRS will not believe deductions which would not have been audited if you had filed on time. Times are tough. The government needs money. New programs to catch non-filers are being tested and implemented.
Most people are wholly in favor of making non-filers pay.
As always, it is better to come forward with back tax returns, rather than wait for the IRS or FTB to knock on your door. California has added some new penalties and costs for non-filers who do not respond for demanded tax returns. This is different from the Federal rules where penalties are based on the net of tax due.