Yes, you can send invoices to clients without ever leaving HubSpot.
Simply create the invoice, sync it to your accounting platform and you'll see a "Send" button appear. You'll be able to check the invoice first, and it'll get sent out using your existing template from Xero or QuickBooks Online.
If you don't want users to be able to send invoices, subscribers on Pro and above can turn on "Finance mode" - which only lets finance teams send invoices.