This pumpkin spice bread and other quick breads like banana bread will fall apart most likely because there is not enough room for it to rise in the pan. The pan is either too small, or it is overfilled with batter. As the batter rises while baking, if there is not enough support around it, it’ll collapse and sink. Be sure to bake the bread in the right pan where it has enough space to rise.