A society without rules is of course impossible, while a society without rulers is not only possible, but has been done several times in several places. If you veered to far from what was acceptable you just knew to leave the village or you would face reprisal. However I think some of the utopian communes of the antebellum North might have been constitutional anarchies. This gets complicated. Rules are not rules unless they can be enforced.
Anarchy is possible, but difficult to maintain.
In fact, the very act of maintaining a state of anarchy involves taking a position of forcing others to live anarchically, an act which destroys the very system it's meant to sustain. The societies I was referring to do not have invisible rulers. Also they don't force people to live in anarchy, I think some leave to enter the wider world. It is just the way their society evolved. In fact I know of no anarchy that has invisible rulers.