In archaeology, carbon dating came from physics. The first airplane was built by the Wright brothers who were bicycle experts. Thus, as an expert in your field, you face a difficult problem. If you listen too much to them then you will not get any of your own work done, but if you ignore them then you may miss your great chance. I have no simple answer except do not dismiss the outsider too abruptly as is generally done by in the insiders.
Brains come in all kinds of flavors.
Experimental physicists do not think the same way as theoreticians do. Some experimentalists seem to think with their hands, i.e., playing with equipment lets them think more clearly. It took me a few years to realize that people who did not know a lot of mathematics still could contribute. Just because they could not solve a quadratic equation immediately in their head did not mean I should ignore them. When someone's flavor of brains does not match yours may be more reason for paying attention to them.