Yeah, ya kinda do. At least, that's my opinion. There's so much texture in the crumble that you really don't need (or want) anymore. The apples are so much tastier when they're nice and tender without a tough skin.