Remove chmod on removed dirs after 04a78191 to prevent Travis-CI errors