There are problems with that approach, but there are certain questions that can be addressed that way. And certain questions that can't. The bigger issue for many researchers is defining and maintaining quality in the midst of all the ferment in their profession. And they are divided over whether setting common standards for education research would help. To some extent, research standards already exist.
Academic journals act as gatekeepers when they send prospective articles out for peer review.
Reviewers also screen the funding proposals that come before the Education Department's office of educational research and improvement. But one recent federal report questions whether the OERI's peer-review panels are doing an adequate job. The report by the National Educational Research Policy and Priorities Board notes that while most peer reviewers appear to be qualified for the task, a few panels that screened proposals for research competitions held in 1996 and 1997 had few or no members with any expertise in research. The Education Department is considering creating standing review panels whose members would work together over longer periods of time, learn from one another, and reach consensus on some common evaluation standards. But I think these things need to be publicly discussed, and the field needs to worry about quality.