Yes, you can leave out the almonds but as always, I highly recommend trying recipes as written before making changes. The toasted almonds in this shrimp salad recipe not only add an amazing crunch, but enhances the flavor as well. Try it before you dismiss using the almonds!