Remove file dir if it is empty.