Return 'BackCompat' for document.compatMode only if in quirksmode

Fixes #2971