Yes, you can: you are under no obligation to limit your use of your land (e.g. have large piles or spillover of dirt) to accommodate your neighbor. You can remove the dirt from your property even if that requires your neighbor to build a retaining wall on his own to put a driveway there. Your property is your property, not his.