detect JRE (vs JDK) and invalid JAVA_HOME

fixes #3739

(cherry picked from commit 9fe99aa6e947a86f4558b56f691e2376da946897)