Remove use of lib-noir

We barely used lib-noir and all its uses could be trivially replaced.
One less dependency = better.