Requiring all of them to be setuid root is a bigger security hole than xterm alone.
Some operating systems require programs to be setuid root in order to manipulate pseudoterminals.
This is as much of a hack as setuid for utmp or load calculations. Fortunately, Linux does not require setuid programs for pseudoterminals. To verify this, I run non-setuid xterms as root and then as kevin, which has no special permissions on my machine. I was able to use the same pseudoterminal as first root and then kevin without any difficulty. The permissions of the pseudoterminal changes, but root permission is not needed.