Add testing for concrete Dir creation failure

Tests that a permission denied error will correctly cause dir creation to fail
with an OSError.