The speaker advises that job hopping early in your career is generally not detrimental and can be beneficial as it allows you to gain diverse experiences. However, as you advance, it becomes more important to stay longer in roles to see projects through and gain deeper experience.