Lahaina, Hawaii
Baldwin was born on August 29, 1842 in Lahaina, Hawaii. His father was American Christian missionary Dwight Baldwin (1798–1886), and his mother was Charlotte Fowler Baldwin. He was named after Matthew LaRue Perrine (1777-1836), professor at Auburn Theological Seminary, from which his father had graduated shortly before his departure to the Hawaiian Islands.