A system for supporting a heterogeneous and asymmetric dual rail configuration in a high performance computing environment, comprising: a computer, the computer comprising one or more microprocessors; a plurality of hosts, each of the plurality of hosts comprising a dual port adapter; a first rail comprising a private Remote Direct Memory Access 'RDMA' fabric, the private RDMA fabric comprising a plurality of switches arranged in two or more racks, each of the two or more racks comprising a set of the plurality of switches; a second rail comprising a public fabric, the public fabric comprising a cloud fabric; and a distributed workload placement orchestrator; wherein the cloud fabric comprises one or more points of delivery 'PODs' where servers are interconnected by the cloud fabric and the private RDMA fabric; wherein each POD of the one or more PODs comprises one or more of racks, servers, switches, networks and infrastructures; wherein each host of the plurality of hosts is connected to the first rail via a first port of the dual port adapter of the host; wherein each host of the plurality of hosts is connected to the second rail via a second port of the dual port adapter of the host; wherein a workload is provisioned at a host of the plurality of hosts; wherein a placement policy is assigned to the provisioned workload; wherein it is determined whether the placement policy of the workload specifies strict placement within a single POD; wherein, if the placement policy specifies strict placement within a single POD, then compute nodes within the single POD are allocated to the provisioned workload, and if the placement policy does not specify strict placement within a single POD, then compute nodes are allocated to the workload without POD consideration; wherein the distributed workload placement orchestrator utilizes the placement policy assigned to the provisioned workload to assign network traffic of the provisioned workload to one or more of the first rail comprising the private RDMA fabric and the second rail comprising the public fabric in accordance with the placement policy; wherein network traffic of the provisioned workload is assigned to the second rail comprising the public fabric when one or more of the following conditions are met: the workload is prohibited from using a private fabric; a target node of the workload belongs to a different POD; the target node is currently not available via private fabric; and current load on the private RDMA fabric is higher than a set threshold and the load on the cloud fabric is lower than another set threshold; and wherein network traffic of the provisioned workload is otherwise assigned to the first rail comprising the private fabric. a computer, the computer comprising one or more microprocessors; a plurality of hosts, each of the plurality of hosts comprising a dual port adapter; a first rail comprising a private Remote Direct Memory Access 'RDMA' fabric, the private RDMA fabric comprising a plurality of switches arranged in two or more racks, each of the two or more racks comprising a set of the plurality of switches; a second rail comprising a public fabric, the public fabric comprising a cloud fabric; and a distributed workload placement orchestrator; wherein the cloud fabric comprises one or more points of delivery 'PODs' where servers are interconnected by the cloud fabric and the private RDMA fabric; wherein each POD of the one or more PODs comprises one or more of racks, servers, switches, networks and infrastructures; wherein each host of the plurality of hosts is connected to the first rail via a first port of the dual port adapter of the host; wherein each host of the plurality of hosts is connected to the second rail via a second port of the dual port adapter of the host; wherein a workload is provisioned at a host of the plurality of hosts; wherein a placement policy is assigned to the provisioned workload; wherein it is determined whether the placement policy of the workload specifies strict placement within a single POD; wherein, if the placement policy specifies strict placement within a single POD, then compute nodes within the single POD are allocated to the provisioned workload, and if the placement policy does not specify strict placement within a single POD, then compute nodes are allocated to the workload without POD consideration; wherein the distributed workload placement orchestrator utilizes the placement policy assigned to the provisioned workload to assign network traffic of the provisioned workload to one or more of the first rail comprising the private RDMA fabric and the second rail comprising the public fabric in accordance with the placement policy; wherein network traffic of the provisioned workload is assigned to the second rail comprising the public fabric when one or more of the following conditions are met: the workload is prohibited from using a private fabric; a target node of the workload belongs to a different POD; the target node is currently not available via private fabric; and current load on the private RDMA fabric is higher than a set threshold and the load on the cloud fabric is lower than another set threshold; and the workload is prohibited from using a private fabric; a target node of the workload belongs to a different POD; the target node is currently not available via private fabric; and current load on the private RDMA fabric is higher than a set threshold and the load on the cloud fabric is lower than another set threshold; and wherein network traffic of the provisioned workload is otherwise assigned to the first rail comprising the private fabric.