block aufs on incompatible file systems

AUFS doesn't work properly on top of btrfs, AUFS and some other file systems. We shouldn't use aufs on these file systems.

Fixes #5900