Yes, it is often illegal. Construction site materials are typically owned by the contractor or property owner. Taking dirt without permission could be considered theft.