For cleaning brick, it is safe to take a cold chisel and hammer and gently tap at the larger part of the concrete. Before doing so, try soaking the concrete with hot soapy water to lubricate it partly. This also works well if the brick or stone is particularly smooth and slick.