Yes, it’s possible to start a business without registering with your state or filing for an EIN from the IRS. However, depending on what you’re doing, you might be required to obtain a proper license.