They certainly can.  The cells that define a mole are called “nevus” cells and like almost any other cell type they can become malignant.  That form of cancer is known as “malignant melanoma.”