The brain gradually shrinks with age, especially as one enters late adulthood and beyond. This natural decline can start as early as our late 20s or 30s. Factors like chronic stress, excessive alcohol consumption, and neurodegenerative diseases like Alzheimer's can also accelerate brain shrinkage.