You can but you'll notice a flavor difference. Bananas that aren't as ripe won't add as much sweetness to the bread.