This a tougher question than it sounds. The simple answer is: The National Fire Protection Association Standard 211 says, “Chimneys, fireplaces, and vents shall be inspected at least once a year for soundness, freedom from deposits, and correct clearances. Cleaning, maintenance, and repairs shall be done if necessary.” This is the national safety standard and is the correct way to approach the problem. It takes into account the fact that even if you don’t use your chimney much, animals may build nests in the flue or there may be other types of deterioration that could make the chimney unsafe to use.
The Chimney Safety Institute of America recommends that open masonry fireplaces should be swept at 1/8″ of sooty buildup, and sooner if there is any glaze present in the system. This is considered to be enough fuel buildup to cause a chimney fire capable of damaging the chimney or spreading to the home. Factory-built fireplaces should be swept when any appreciable buildup occurs. The logic is that the deposit is quite acidic and can shorten the life of the fireplace.