Isn’t Ijma And Tawatur Just Opinions?
Also the underlying principle would be that the absence of evidence does not imply evidence of absence. Just because we don’t find dissenting opinions on eg there being 5 prayers, doesn’t mean that people never held those opinions
If rituals would have drifted away there would be proof evident from different people following different things. There are of course such practices, e.g. rafa yadein, and that’s why they’re not considered sunnah.