Re-enable asserts, but only for non-directory files