The tree symbolizes the speaker’s, first love. It grew slowly yet steadily. But the speaker failed to notice/recognize its growth.