No. A scammer will not be able to withdraw money from your account if they only have your account number. If they have enough personal information, they may be able to steal your identity and perform fraudulent acts in your name.