If you're going to rebind System/setErr you should record the value it was and put it back in a finally block.  Also, /dev/null isn't portable, you should just create a PrintStream on ByteArrayOutputStream instead.