Almonds with or without skin won’t make a massive difference in the taste or texture of the gravy. Hence, I would suggest you let the skin of the almonds be as it is.