Remove the build directory in buildsrc for the gradle plugin