Fix the issue that ArgumentError has raised when calling destroy_all to associated member