Detect TTY support and avoid ANSI colors if none

fixes #5992