You can still play through the third semester without doing this, but there are some extra scenes and important context that you'll miss out on. Akechi is a bit harder to hang out with than the other Phantom Thieves, as his schedule can be a bit weird.