Yes, it's perfectly okay to decline if it interferes with your personal commitments. Politely explain your situation and express your appreciation for the invitation.