Both are correct. Beware of the dog refers to a specific dog. Beware of dog is just a general warning that a dog (or dogs) are present.