This type of fitness won't disappear in a matter of days, but it will decrease over time, according to Dr. Campbell. Generally, he says, cardiovascular fitness starts to noticeably decrease after about two weeks of being sedentary.