Remove the ignored directories from repo