Fix bug when only owner or group needs to change on directories