The biggest culprit of sinking banana bread is taking it out of the oven before it is finished. If the center of the loaf is still raw or too soft, it will collapse as it cools. Make sure to cook it until a toothpick inserted in the middle comes out clean or with a few large crumbs.