It doesn't seem fair, but the fitter you are, the faster you lose the benefits of regular exercise. On the other hand, fitter people can afford to lose a lot more than the less fit. Loss of fitness occurs the fastest during the first three weeks of down time.
Exercise challenges the body and it responds by adapting to meet the challenge.
When you discontinue your workouts, your body responds by adapting back down to the lower level of challenge you're placing on it. In the absence of exercise, the psychological effects of regular exercise begin to wane almost immediately. This happens because during exercise the body produces hormones such as adrenaline and endorphins. These substances are mood elevators. Unfortunately, it takes just a couple of days of slacking for these mood enhancing substances to start dropping.