You will want to take your time and approach this project with care. None of it should harm the fireplace, but if you jam the wire brush or putty knife in at the wrong angle, you could inadvertently damage the brick.