Yes, you can skip the banana. Just be aware that it will be less thick and creamy. Avocado makes a good substitute.