In most cases, this absorbed energy becomes thermal energy in the cloth. Because of this extra thermal energy, the cloth's temperature rises and it begins to transfer the thermal energy to its surroundings as heat. Its temperature stops rising when the thermal energy it receives from the light is exactly equal to the thermal energy it transfers to its surroundings as heat. A cloth's color is determined by how it absorbs and emits light. Black cloth absorbs essentially all light that hits it, which is why its temperature rises so much.
White cloth absorbs virtually no light, which is why it remains cool.
Colored cloths fall somewhere in between black and white. Blue cloth absorbs light in the green and red portions of the spectrum while reflecting the blue portion. Red cloth absorbs light in the blue and green portions of the spectrum while reflecting the red portion. Since most light sources put more energy in the red portion of the spectrum than in the blue portion of the spectrum, the blue cloth absorbs more energy than the red cloth. So the sequence of temperatures you observed is the one you should expect to observe.