You have to be dating first and then ask him to join you during the Firefly Festival. To be dating select L or R while talking to him and tell him that you love him. There is a chance he'll say yes or no but if he says yes you can then start dating :-D! Then ask him out to the Firefly Festival.