Fix the remove source directory mess made by ExternalProject