Ideally, yes. Buying organic is a great way to reduce your exposure to toxins. But you don’t have to be perfect, every little bit helps! To determine the best bang for your buck use the Dirty Dozen / Clean Fifteen list as determined by the Environmental Working Group (EWG).
Buying local produce is also a good strategy because it ensures food is as fresh as possible. This is when nutrient content is at its highest.   Always remember to wash your fruits and veggies well, whether organic or not!