Add error when running overlay over btrfs.

Addresses #9820.