While it's true that castors can damage carpet, it's not necessarily the case that they will always do so. It really depends on the type of castor and the material of your carpet. For example, if you have a hardwood or tile floor, then a castor with a soft rubber wheel is unlikely to cause any damage. However, if you have a delicate carpet made of natural fibers, then a metal or plastic wheeled castor could potentially snag and tear the fibers.
The best way to avoid damaging your carpet with castors is to choose ones with smooth, soft wheels that are designed for use on carpets. You should also make sure that the weight of whatever you're rolling on the caster is evenly distributed. Avoid putting all the weight on just one caster, as this can cause indentations in your carpet where the wheel presses down. If possible, try to roll something backwards and forwards rather than side to side to minimize friction and wear on your carpet fibers.