Many people like to do this especially in hot climates or if using the rollers after other treatments that inflame the skin.
Refrigeration will do no harm to the rollers and can be refreshing.
Generally however this is unnecessary crystals like jade are famous for the fact they stay cool even when in contact with the skin for prolonged periods.