You can still install it over drywall with finishing nails so it won't damage the sheetrock underneath. When you remove the brick panels later, just patch the finishing nail holes with spackle.