To determine the right time for harvesting avocados, consider the variety-specific maturity cues, such as changes in color and size. Additionally, perform the "hand test" by gently squeezing the fruit to check for ripeness. Harvesting too early or too late can affect the taste and quality of avocados.