Yes. The relay client makes connections to the Azure Relay service by using fully qualified domain names. Customers can add an entry for *.servicebus.windows.net on firewalls that support DNS approval listing.
It's also possible to allowlist a specific namespace by using your-namespace-name.servicebus.windows.net. In this case you'll also need to allowlist the gateways for your namespace, which can be found by using this PowerShell script.