If you are talking about hanging out with Sumire, specifically, rather than Kasumi, this can only happen in the third semester of the game. During this time, if you head to the new Kichijoji area anytime after school, you can hang out with her. The only exceptions are Sundays and days when it is snowing. In addition, you need to have a Faith Persona in your group when you do this.