Both. If you fry potatoes in oil, then you are officially making patates frites. So patat refers to potatoes and frites refers to frying. Broadly speaking, people in the Netherlands above the great rivers speak of a patatje and people from below the rivers speak of a fritje. In the South, "patat" was already a name for the common potato, therefore it did not qualify as a shortened name for patates frites.