For the interior, this is a style decision, while on the exterior, it should be replaced with the siding or when it softens and begins to rot. If you can push a screwdriver into your trim, it should be replaced.