They had been angry about his indifference. They had felt abandoned by their Lord. They had scolded him for not caring. Why were they afraid and he rebuked them for their lack of faith. Fear is an emotion.
Fear is neither good nor bad, it just is.
You can't stop feeling fearful. You just are. You fear that someone may break into your house, so you get a furious guard dog to protect your home and family, but you are still fearful. But you are still fearful. You grew up in an unstable home or a poor home and you never want to go back to living like that.