Yes, there is a possibility that fraudsters can withdraw money from your bank account if they obtain your essential details such as bank account and routing number.