Well, for this rustic cake, you should. It does not matter how thin the apples' skin is; it will always be harder than the insides.  So, you have to remove the skin so the apples will soften more, making every bite better.