better message when notebook format is not supported

Different messages for bad version and bad JSON.

closes #1592
Merge pull request #3741 from minrk/badnbmsg

better message when notebook format is not supported

different messages for bad version and bad JSON.

closes #1592