Patent Description:
A single-root input/output virtualization (Single-Root Input/Output Virtualization, SR-IOV) technology proposed by the Peripheral Component Interconnect Special Interest Group (Peripheral Component Interconnect Special Interest Group, PCI-SIG) is applied to a scenario of sharing an input/output (Input/Output, I/O) device by multiple processes in a host. The SR-IOV technology virtualizes, for a PCI Express (PCI express, PCIe) device, multiple virtual functions (Virtual Function, VF) for an upper-layer host to invoke.

A multi-root input/output virtualization (Multi-Root Input/Output Virtualization, MR-IOV) technology is applied to a scenario of sharing an I/O device by multiple hosts. The technology virtualizes multiple PCI virtual hierarchies (Virtual Hierarchies, VH) to be invoked by the multiple hosts. However, the technology modifies a protocol above a data link layer in a PCIe protocol. Therefore, multiple nodes, for example, a root port (Root Port, RP) supporting MR-IOV, a switch (Switch), or a PCIe device, in a topology structure need to perform adaption. The MR-IOV technology requires industry chain adaption, causing poor compatibility. At present, there is no product supporting the standard.

In the prior art, a solution of sharing an I/O device by multiple hosts by using an SR-IOV device is provided. <FIG> shows a typical implementation in the prior art. The system <NUM> includes: N hosts (from a host <NUM> to a host N) <NUM>, a PCIe switch <NUM>, M I/O devices (from an I/O device <NUM> to an I/O device M) <NUM>, an external management CPU <NUM>, and a memory <NUM>, where <NUM>≤M≤N. The management CPU <NUM> is responsible for: performing device enumeration and discovery on an I/O device side, and performing device configuration and management. The PCIe switch <NUM> includes a non-transparent link (Non-Transparent Link, NT-L) module, multiple PCIe upstream ports (Upstream Port, UP), a global non-transparent (Non-Transparent, NT) EP module, and multiple virtual PCIe downstream ports (Downstream Port, DP). The global NT EP module is configured to: manage a register and map a memory address, and implement a non-transparent virtual (Non-Transparent Virtual, NT-V) function. The system <NUM> may include multiple virtual hierarchies (Virtual Hierarchy, VH).

In the technical solution, a management CPU needs to be additionally disposed, a memory corresponding to the management CPU needs to be externally disposed, and a separate PCIe management interface needs to be provided. Consequently, the technical solution is not suitable for a product layout. In addition, in the system <NUM> shown in <FIG>, only one management CPU is disposed for managing the entire system, and there is only one management link. Once the management link has an exception, an entire network is broken down and cannot be used. Consequently, stability and reliability are relatively poor.

<CIT> discloses a method for implementing a zoning protocol for DFCF on distributed Ethernet switches. Particularly, it discloses switches, such as distributed network switch or switch modules. Both, distributed network switch and the switch modules can connect servers through network. <CIT>) discloses PCIe switches (i.e., tunnels) where each of these tunnels has a PCIe port and a non-PCIe port, such that each tunnel can convert a PCIe packet into a non-PCIe packet or vice versa.

The present invention is defined by the system of independent claim <NUM> and by the system of independent claim <NUM>. In the following, parts of the description and drawing referring to embodiments, which are not covered by the claims are not presented as embodiments of the invention, but as examples useful for understanding the invention.

The invention made is disclosed in the embodiment referring to <FIG>.

The following describes the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention.

<FIG> is a schematic architectural diagram of a PCIe system <NUM> according to an embodiment of the present invention. The PCIe system <NUM> may be configured to implement sharing an I/O device by multiple processes in one host or implement sharing an I/O device by multiple hosts.

As shown in <FIG>, the PCIe system <NUM> includes: N<NUM> hosts <NUM>, a switching device <NUM>, and M<NUM> I/O devices <NUM>, where N<NUM>≥<NUM>, and M<NUM>≥<NUM>. The I/O device may also be referred to as an endpoint (Endpoint, EP) device.

<FIG> shows, by using an example, that the PCIe system <NUM> includes only one switching device, and the switching device is connected to at least one host and at least one I/O device. However, it should be understood that the PCIe system <NUM> may alternatively include multiple switching devices, and each switching device may be connected to at least one host and at least one I/O device. Each switching device and at least one host and at least one I/O device that are connected to the switching device may be considered as a network unit. Correspondingly, the PCIe system <NUM> may include one or more network units. However, this is not limited in this embodiment of the present invention.

Optionally, in this embodiment of the present invention, the host may be specifically a device such as a server or a personal computer. Alternatively, the host may be a virtual machine. This is not limited in this embodiment of the present invention. Optionally, some or all of the N<NUM> hosts may include a baseboard management controller (Baseboard Management Controller, BMC), that is, the N<NUM> hosts may include one or more BMCs. At least one of the one or more BMCs may have a configuration management function, and is specifically configured to manage and control the network unit including the at least one host, the switching device, and the at least one I/O device. For ease of description, a BMC having a configuration management function is referred to as a management BMC in the following description. Optionally, the management BMC may be configured to implement a configuration management function possessed by a management CPU in the prior art. However, different from the management CPU in the prior art, the management BMC may be responsible for only configuration management of a switching device, and at least one host and at least one I/O device that are connected to the switching device in a network unit to which the management BMC belongs, but not be responsible for configuration management of hosts, I/O devices, and switching devices in other network units that may exist in a network. This is not limited in this embodiment of the present invention.

Optionally, the I/O device may be specifically a device such as a host bus adapter (Host Bus Adapter, HBA) or a solid state disk (Solid State disk, SSD). This is not limited in this embodiment of the present invention.

In this embodiment of the present invention, the switching device <NUM> may include N<NUM> PCIe upstream ports <NUM>, M<NUM> PCIe downstream ports <NUM>, and an internal processing apparatus <NUM>, where N<NUM>><NUM>, and M<NUM>≥<NUM>. The Nz PCIe upstream ports <NUM> are configured to connect to the N<NUM> hosts <NUM>. The M<NUM> PCIe downstream ports <NUM> are configured to connect to the M<NUM> I/O devices <NUM>. The internal processing apparatus <NUM> may be connected to the M<NUM> PCIe downstream ports <NUM> by using an internal connection line of the switching device, and may be connected to the N<NUM> PCIe upstream ports <NUM> by using the internal connection line of the switching device.

Specifically, the switching device <NUM> may be connected to the N<NUM> hosts <NUM> by using the N<NUM> PCIe upstream ports <NUM>. If N<NUM>=N<NUM>, that is, a quantity of the PCIe upstream ports of the switching device is equal to a quantity of the hosts connected to the switching device, the Nz PCIe upstream ports may be connected to the N<NUM> hosts in a one-to-one correspondence. If N<NUM>>N<NUM>, that is, a quantity of the PCIe upstream ports of the switching device is greater than a quantity of the hosts connected to the switching device, N<NUM> of the N<NUM> PCIe upstream ports may be connected to the N<NUM> hosts in a one-to-one correspondence, and remaining (N<NUM>-N<NUM>) PCIe upstream ports may be idle, or may be set as PCIe downstream ports. However, this is not limited in this embodiment of the present invention. Similarly, the switching device <NUM> may be connected to the M<NUM> I/O devices <NUM> by using the M<NUM> PCIe downstream ports <NUM>. If M<NUM>=M<NUM>, that is, a quantity of the PCIe downstream ports of the switching device is equal to a quantity of the I/O devices connected to the switching device, the M<NUM> PCIe downstream ports may be connected to the M<NUM> I/O devices in a one-to-one correspondence. If M<NUM>>M<NUM>, that is, a quantity of the PCIe downstream ports of the switching device is greater than a quantity of the I/O devices connected to the switching device, M<NUM> of the M<NUM> PCIe downstream ports may be connected to the N<NUM> I/O devices in a one-to-one correspondence, and remaining (M<NUM>-M<NUM>) PCIe downstream ports may be idle or may be set to PCIe upstream ports. If M<NUM><M<NUM>, that is, a quantity of the PCIe downstream ports of the switching device is less than a quantity of the I/O devices connected to the switching device, the PCIe system <NUM> may further include one or more PCIe switches, and some or all of the M<NUM> PCIe downstream ports may be connected to the M<NUM> I/O devices by using the PCIe switch. In this way, the quantity of the I/O devices connected to the switching device may be increased, and switching performance is not reduced. However, this is not limited in this embodiment of the present invention. Optionally, the PCIe system <NUM> further includes the PCIe switch, and the M<NUM> PCIe downstream ports may be connected to the M<NUM> I/O devices by using the PCIe switch.

In addition, the switching device <NUM> further includes the internal processing apparatus <NUM>. The internal processing apparatus <NUM> may be specifically a processing module, for example, a processor or/or a hardware processing circuit. Optionally, the internal processing apparatus may be implemented by a processor and a hardware circuit. Alternatively, the internal processing apparatus may be implemented only by a hardware circuit. This is not limited in this embodiment of the present invention. In this embodiment of the present invention, the internal processing apparatus may be configured to enumerate a physical device (that is, a real device, including no functional module) under the switching device, for example, determine a physical device connected to the switching device by using the M<NUM> PCIe downstream ports <NUM>, or may further obtain information, for example, an identification (a BDF) or a topology structure, of multiple physical devices connected to the switching device, and may further set up a PCIe domain structure tree according to the information about the multiple physical devices. The physical device may include the I/O device and/or the PCIe switch. In this case, the internal processing apparatus may be considered as a virtual root port (Root Port, RP) or a root complex (Root Complex, RC). However, this is not limited in this embodiment of the present invention.

Optionally, the internal processing apparatus <NUM> may actively perform device enumeration, for example, perform device enumeration at a system initialization stage. Alternatively, the internal processing apparatus <NUM> may perform, when receiving an instruction of another device, device enumeration according to the instruction. For example, the internal processing apparatus <NUM> may receive an initialization instruction or an enumeration instruction that is sent by the management BMC, and perform device enumeration according to the initialization instruction or the enumeration instruction. However, this is not limited in this embodiment of the present invention.

Specifically, when performing device enumeration, the internal processing apparatus <NUM> may send multiple configuration read/write packets to the at least one PCIe downstream port <NUM>. The configuration read/write packet carries a request identification (Request Identification, RID) and a completer identification (Completer identification, CID), where a value of the RID may be set to a BDF of the internal processing apparatus <NUM>, and a value of the CID may be numbered in sequence starting from an initial value. The internal processing apparatus <NUM> may receive a configuration read/write response packet corresponding to at least one of the multiple configuration read/write packets. The configuration read/write response packet carries a RID and a CID in the corresponding configuration read/write packet. In this way, the internal processing apparatus <NUM> determines, by recognizing the configuration read/write response packet (for example, recognizing the CID in the configuration read/write response packet), whether the switching device <NUM> is connected to a physical device whose BDF number is the CID in the configuration read/write response packet. However, this is not limited in this embodiment of the present invention.

In an optional embodiment, if the internal processing apparatus <NUM> is implemented by a processor and a hardware circuit, where the processor and the hardware circuit may be connected by using an internal bus, the processor may be configured to: receive an initialization instruction (or an enumeration instruction) from the management BMC; generate, according to the initialization instruction, information that is about an enumeration packet and that includes information needed by configuration read/write packets of type <NUM> (Type0) and type <NUM> (Type <NUM>); and output the generated information to the internal bus. The hardware circuit may receive, by using the internal bus, the information generated by the processor, encapsulate the information as a transaction layer packet (Transaction Layer Packet, TLP), and send the TLP to the at least one PCIe downstream port <NUM> by using a next-level module or circuit connected to the internal processing apparatus <NUM>. In addition, the hardware circuit may be further configured to recognize a configuration read/write response packet when receiving the configuration read/write response packet sent by the at least one PCIe downstream port <NUM>. For example, the hardware circuit may receive a completion (Completion, CPL)/(Completion Data, CPLD) packet transmitted by the at least one PCIe downstream port <NUM>, and recognize an ID in the received CPL/CPLD packet. By means of interruption trigger of the hardware circuit, the CPL/CPLD packet may be transmitted to the processor of the internal processing apparatus <NUM> by using the internal bus. Specifically, the hardware circuit may disassemble the CPL/CPLD packet into a data format complying with a time sequence of the internal bus, and send the CPL/CPLD packet to the internal bus. The processor of the internal processing apparatus <NUM> may be further configured to: parse received packet information, store the packet information, and generate a structure tree of a PCIe domain according to the packet information. After the device enumeration is completed, the processor may further transmit information about the generated PCIe structure tree to the management BMC by using the multiple PCIe upstream ports <NUM>. However, this is not limited in this embodiment of the present invention. Therefore, the PCIe system provided in this embodiment of the present invention includes the at least one host, the switching device, and the at least one I/O device. The switching device includes the multiple PCIe upstream ports configured to connect to the at least one host, the at least one PCIe downstream port configured to connect to the at least one I/O device, and the internal processing apparatus. The processing apparatus is connected to the at least one PCIe downstream port by using an internal bus, and is configured to: transmit a configuration read/write packet to the at least one PCIe downstream port by using the internal connection line, receive a configuration read/write response packet transmitted by the at least one PCIe downstream port by using the internal connection line, and determine, according to a completer identification carried in the configuration read/write response packet, that the switching device is connected to an I/O device whose identification is the completer identification. In this way, when a switching device in the PCIe system is faulty, only at least one host and at least one I/O device that are connected to the switching device may be affected, and another switching device, and a host and an I/O device that are connected to the another switching device in the PCIe system are not affected. Therefore, compared with an external management CPU in the prior art, stability and reliability of the PCIe system can be improved.

Optionally, the internal processing apparatus <NUM> may further have at least one of the following functions: hot swap event processing, exception event processing, or component configuration (for example, register configuration).

The internal processing apparatus <NUM> may be configured to process a hot swap event. Specifically, the internal processing apparatus <NUM> may detect a hot swap event triggered by the management BMC and/or hardware, and perform a corresponding hot swap processing procedure. The internal processing apparatus <NUM> may be further configured to process an exception event. Specifically, the internal processing apparatus <NUM> may process a hardware exception, record an exception event log, and report the exception event log. In addition, the internal processing apparatus <NUM> may be further configured to configure a component. Specifically, the internal processing apparatus <NUM> may configure a register at an initialization stage, or modify a configuration of the register. The internal processing apparatus <NUM> may obtain information needed for the component configuration in a process of device enumeration, or may receive configuration management information sent by the management BMC, configure the component according to the configuration management information, and the like. This is not limited in this embodiment of the present invention.

In an optional embodiment, if the internal processing apparatus <NUM> is implemented by the processor and the hardware circuit, when implementing the function of exception event processing, the hardware circuit in the internal processing apparatus <NUM> may capture exception information related to a physical function of the I/O device, and trigger the processor by using interruption information configured by the processor in the internal processing apparatus <NUM>. The processor of the internal processing apparatus <NUM> may recognize an exception type. If the exception type is specifically a correctable or an uncorrectable nonfatal error, the processor may perform self-processing, and output only an alarm without reporting. If the exception type is specifically an uncorrectable fatal error, the processor may break a link of the I/O device, and report the broken link of the I/O device to the management BMC. After learning the I/O device whose link is broken, the management BMC may uninstall a drive corresponding to the I/O device. In addition, optionally, the management BMC may notify a shelf management module (shelf management module, SMM) of update of the networking information. The SMM manages and controls a host corresponding to the I/O device having an exception, for example, performs drive uninstallation or resets initialization processing. This is not limited in this embodiment of the present invention.

In this embodiment of the present invention, the internal processing apparatus <NUM> may be configured to: receive a data packet sent by the M<NUM> I/O devices <NUM> by using the M<NUM> PCIe downstream ports <NUM>, process the received data packet, for example, convert a time sequence, and send the processed data packet to the N<NUM> PCIe upstream ports <NUM> by using the internal connection line. In addition, the internal processing apparatus <NUM> may be further configured to: receive a data packet sent by the N<NUM> hosts <NUM> by using the N<NUM> PCIe upstream ports <NUM>, process the received data packet, for example, convert a time sequence, and send the processed data packet to the M<NUM> PCIe downstream ports <NUM> by using the internal connection line.

In an optional embodiment, if the internal processing apparatus <NUM> is implemented by the processor and the hardware circuit, when receiving data sent by the host, the processor in the internal processing apparatus may parse a bus time sequence of an interface of the management BMC, and output the data in a time sequence of the internal bus. When receiving data forwarded by a next-level module or circuit, the processor may parse the time sequence of the internal bus, and output the data to the bus time sequence of the interface of the management BMC. However, this is not limited in this embodiment of the present invention.

Optionally, in this embodiment of the present invention, the management BMC may report networking information (or networking configuration information) to the SMM by using a network port. The SMM may be configured to be responsible for managing an entire network. A user may set, according to a requirement, a mapping connection between the host and the I/O device by using a human-computer interaction interface of the SMM. In this case, optionally, the management BMC may be further configured to: receive configuration requirement information sent by the SMM, and determine the configuration management information according to the configuration requirement information. However, this is not limited in this embodiment of the present invention.

Optionally, in this embodiment of the present invention, each of the Nz PCIe upstream ports <NUM> has a PCIe configuration space, and none of the M<NUM> PCIe downstream ports <NUM> has any PCIe configuration space.

In this embodiment of the present invention, the PCIe upstream port <NUM> in the switching device <NUM> has the PCIe configuration space, and is a standard PCIe port (that is, a standard P2P upstream bridge). The PCIe downstream port <NUM> in the switching device <NUM> has no PCIe configuration space, and is not a standard PCIe port (that is, is not a standard P2P downstream bridge). Optionally, the switching device <NUM> may not additionally store PCIe configuration spaces of the M<NUM> PCIe downstream ports <NUM>, that is, there is no PCIe configuration space of the PCIe downstream port <NUM> in the switching device <NUM>. Alternatively, the switching device <NUM> may additionally store a PCIe configuration space of each of the M<NUM> PCIe downstream ports <NUM>, that is, the PCIe configuration space of the PCIe downstream port <NUM> is separated from the PCIe downstream port <NUM> in the switching device <NUM>. Alternatively, the switching device <NUM> may additionally store PCIe configuration spaces of some of the M<NUM> PCIe downstream ports <NUM> but not additionally store PCIe configuration spaces of some other PCIe downstream ports <NUM>. This is not limited in this embodiment of the present invention.

Optionally, in this embodiment of the present invention, the PCIe downstream port <NUM> may correspond to at least one PCIe configuration space, and the at least one PCIe configuration space corresponding to the PCIe downstream port <NUM> and the PCIe downstream port <NUM> may be individually disposed in the switching device <NUM>, that is, the PCIe downstream port <NUM> may be separated from the at least one PCIe configuration space corresponding to the PCIe downstream port <NUM>. In addition, optionally, the at least one PCIe configuration space corresponding to the PCIe downstream port <NUM> may belong to a PCIe domain corresponding to the N<NUM> hosts <NUM>. In this way, when the switching device <NUM> stores multiple PCIe configuration spaces corresponding to a PCIe downstream port <NUM>, the PCIe downstream port <NUM> may be used as multiple ports to be used by the hosts <NUM>, so as to improve resource utilization of the system.

It should be understood that an example in which the N<NUM> PCIe upstream ports <NUM> in the switching device <NUM> all have the PCIe configuration space and the M<NUM> PCIe downstream ports <NUM> all have no PCIe configuration space is used for description in the foregoing embodiment. Optionally, some of the M<NUM> PCIe downstream ports <NUM> may have a PCIe configuration space, and some other PCIe downstream ports <NUM> may have no PCIe configuration space. Alternatively, all of the M<NUM> PCIe downstream ports <NUM> have a PCIe configuration space. In another optional embodiment, some or all of the Nz PCIe upstream ports <NUM> may have no PCIe configuration space. This is not limited in this embodiment of the present invention.

Optionally, as shown in <FIG>, the switching device <NUM> further includes: N<NUM> virtual endpoint device modules (vEPs for short) <NUM>, a mapping module (MAP) <NUM>, and M<NUM> mirroring endpoint device modules (mEPs for short) <NUM>.

The N<NUM> vEPs <NUM> may be connected to the N<NUM> PCIe upstream ports <NUM>, and are configured to virtualize functions of the M<NUM> I/O devices <NUM> connected to the M<NUM> PCIe downstream ports <NUM>, so that the functions are used by the N<NUM> hosts <NUM> connected to the N<NUM> PCIe upstream ports <NUM>, where N<NUM>≥<NUM>.

The M<NUM> mEPs <NUM> may be connected to the M<NUM> PCIe downstream ports <NUM>, and are configured to store PCIe configuration content of the M<NUM> I/O devices <NUM> connected to the M<NUM> PCIe downstream ports <NUM>, where M<NUM>≥<NUM>.

The mapping module <NUM> may be separately connected to the N<NUM> vEPs <NUM> and the M<NUM> mEPs <NUM>, and is configured to implement mapping between a PCIe domain corresponding to the N<NUM> hosts <NUM> and a PCIe domain corresponding to the M<NUM> I/O devices <NUM>.

In this embodiment of the present invention, optionally, the switching device <NUM> may further include one or more mEPs <NUM>, the mapping module <NUM>, and one or more vEPs <NUM>. A value of N<NUM> may be determined by a quantity of functions that need to be used or can be used by the N<NUM> hosts <NUM>, and N<NUM> may be equal to or not equal to M<NUM>. For example, N<NUM> and M<NUM> may both be equal to a total quantity of functions (for example, a total quantity of VFs) possessed by the M<NUM> I/O devices <NUM>. This is not limited in this embodiment of the present invention.

Specifically, the mEP <NUM> may be mirroring of a real EP, and is configured to store the PCIe configuration content of the I/O device (that is, the endpoint device) <NUM>. The vEP <NUM> may be configured to virtualize a physical function and/or a virtual function of the endpoint device <NUM>, and may be specifically a PCIe configuration space corresponding to the physical function or the virtual function of the endpoint device <NUM>. The mapping module <NUM> may be configured to implement mapping between identifications and/or addresses of the PCIe domain corresponding to the N<NUM> hosts <NUM> and the PCIe domain corresponding to the at least one I/O device <NUM>, to determine a module (or component) to which received information and/or data is sent. In an optional example, the vEP <NUM> may belong to the PCIe domain corresponding to the host <NUM>, and the mEP <NUM> may belong to the PCIe domain corresponding to the I/O device <NUM>. Correspondingly, the mapping module <NUM> may be specifically configured to implement mapping between an identification and/or address of the M<NUM> mEPs <NUM> and an identification and/or address of the N<NUM> vEPs <NUM> (that is, perform mapping from the vEPs to the mEPs or perform mapping from the mEPs to the vEPs), so as to perform forwarding processing on a data packet transmitted between the N<NUM> hosts <NUM> and the M<NUM> I/O devices <NUM>. However, this is not limited in this embodiment of the present invention.

Optionally, the Nz PCIe upstream ports <NUM>, the N<NUM> vEPs <NUM>, the mapping module <NUM>, the M<NUM> mEPs <NUM>, and the M<NUM> PCIe downstream ports <NUM> may be connected in sequence. Specifically, one end of the vEP <NUM> may be connected to the PCIe upstream port <NUM>, and another end may be connected to the mapping module <NUM>. The N<NUM> vEPs <NUM> may be directly or indirectly connected to some or all of the Nz PCIe upstream ports <NUM>. One end of the mEP <NUM> may be connected to the PCIe downstream port <NUM>, and another end may be connected to the mapping module <NUM>. The M<NUM> mEPs <NUM> may be directly or indirectly connected to some or all of the M<NUM> PCIe downstream ports <NUM>. However, this is not limited in this embodiment of the present invention.

If the PCIe configuration spaces of the M<NUM> PCIe downstream ports <NUM> are not additionally set in the switching device <NUM>, the N<NUM> vEPs <NUM> may be directly connected to some or all of the Nz PCIe upstream ports <NUM>. Optionally, if the switching device <NUM> further additionally stores a PCIe configuration space corresponding to one or more PCIe downstream ports <NUM>, for example, as shown in <FIG>, the switching device <NUM> may store N<NUM> PCIe downstream port configuration spaces (DP_CFG for short) <NUM>, each PCIe downstream port <NUM> may correspond to zero, one, or more pieces of DP_CFG <NUM>. In this case, the N<NUM> vEPs <NUM> may be connected to the N<NUM> PCIe upstream ports <NUM> by using the N<NUM> pieces of DP_CFG <NUM>. In this way, the PCIe upstream port <NUM> is connected to the vEP <NUM> by using the PCIe configuration space corresponding to the PCIe downstream port <NUM>, so as to increase a quantity of vEPs that can be used by each host, thereby enhancing system performance.

In addition, in this embodiment of the present invention, a virtual function is presented to the host, so that the PCIe system supports any Windows version to use the virtual function in the host.

In an optional embodiment, the mapping module <NUM> stores:.

Optionally, the first mapping table may be specifically used to store a mapping relationship from an identification of the N<NUM> vEPs <NUM> to an identification of the M<NUM> mEPs <NUM>, for example, a mapping relationship from a BDF number of at least one vEP <NUM> to a BDF number of at least one mEP <NUM>. The second mapping table may be specifically used to store a mapping relationship from the identification of the M<NUM> mEPs <NUM> to the identification of the N<NUM> vEPs <NUM>. However, this is not limited in this embodiment of the present invention.

Optionally, in this embodiment of the present invention, the mapping module <NUM> may be implemented by a register and a hardware processing circuit. In this case, the first mapping table and the second mapping table may be separately stored by the register. In addition, optionally, the identifications in the first mapping table and the second mapping table may be represented by BDFs. Alternatively, a function of the I/O device <NUM> or a function of the vEP <NUM> may be renumbered, for example, be numbered in sequence starting from <NUM>, to obtain an index of each function, and the index is stored in the mapping tables in a function index form, to save storage space.

Table <NUM> and Table <NUM> respectively show possible implementations of the first mapping table and the second mapping table. As shown in Table <NUM>, the first mapping table may include four columns. A value in a column of "function index" indicates an index of a function in the PCIe domain corresponding to the I/O device <NUM>. The index may be obtained by numbering BDFs of all functions, and may occupy one or several bits. A value in a column of "enable indication" indicates whether a function corresponding to the function index exists. An mEP bus number and an mEP function number may respectively indicate a bus number and a device function number of an mEP <NUM> corresponding to the function index. The mEP bus number and the mEP function number are combined into a BDF number of the mEP <NUM>. The BDF number of the mEP <NUM> may be associated with a BDF number of an I/O device <NUM> corresponding to the mEP <NUM>. A quantity of rows included in the first mapping table is not limited in this embodiment of the present invention. For example, the first mapping table may include <NUM> entries (entry). However, this is not limited in this embodiment of the present invention.

As shown in Table <NUM>, the second mapping table may include five columns. A value in a column of "function index" indicates an index (for example, a function index of the vEP <NUM>) of a function in the PCIe domain (which may be a virtual functional module and/or physical device discovered by the host <NUM> by means of enumeration) corresponding to the host <NUM>. The index may be obtained by numbering BDFs of all functions, and may occupy one or several bits. A value in a column of "enable indication" indicates whether a function corresponding to the function index exists. An eVF bus number and an eVF device function number respectively indicate a bus number and a device function number of an eVF <NUM> corresponding to the function index. The eVF bus number and the eVF device function number are combined into a BDF of the eVF <NUM>. A value in a column of "PE indication" is used to indicate whether the function corresponding to the function index is a vF or a PF. A quantity of rows included in the second mapping table is not limited in this embodiment of the present invention. For example, the second mapping table may include <NUM> entries (entry). However, this is not limited in this embodiment of the present invention.

In this way, the mapping relationship between the identification in the PCIe domain corresponding to the host <NUM> and the identification in the PCIe domain corresponding to the I/O device <NUM> is stored in a mapping table manner. Compared with the prior art, this solution can reduce storage space occupied by the mapping module <NUM> and complexity, and save a system storage resource. In addition, the BDFs of all the functions are numbered and the numbers are stored in the mapping table. Compared with storing a BDF of <NUM> bits, the storage space occupied by the mapping module <NUM> can be further reduced. Optionally, in this embodiment of the present invention, the PCIe configuration content, of the endpoint device <NUM>, stored in the mEP <NUM> may be specifically a PCIe configuration space of the endpoint device <NUM>, or may be part configuration content in a PCIe configuration space of the endpoint device <NUM>. For example, all of the M<NUM> mEPs <NUM> may be specifically mapping tables. Alternatively, some of the M<NUM> mEPs <NUM> may be specifically mapping tables. This is not limited in this embodiment of the present invention.

In an optional embodiment, a first mEP in the M<NUM> mEPs <NUM> is specifically a third mapping table. The third mapping table is used to store a base address register (Base Address Register, BAR) address and a BAR size of a virtual function of a first I/O device in the M<NUM> I/O devices. The first mEP is configured to store configuration content of the first I/O device. Table <NUM> shows a possible implementation of the third mapping table. The third mapping table includes an mEP index and a BAR address and a BAR size that correspond to the mEP index. Optionally, each mEP may include six BARs. However, this is not limited in this embodiment of the present invention.

In this case, the mapping module <NUM> may implement the mapping from the address in the PCIe domain of the host <NUM> to the address in the PCIe domain of the I/O device <NUM> with reference to the third mapping table and the first mapping table. However, this is not limited in this embodiment of the present invention.

In this way, address information of the virtual function of the I/O device is stored in a mapping table manner. Compared with the prior art, this solution can reduce storage space occupied by the mEP, and save a system storage resource.

It should be understood that examples of Table <NUM> to Table <NUM> are intended to help a person skilled in the art better understand this embodiment of the present invention, but not to limit the scope of this embodiment of the present invention. Apparently, a person skilled in the art may perform various equivalent modifications or changes according to the provided examples of Table <NUM> to Table <NUM>. The modifications or the changes also fall into the scope of this embodiment of the present invention.

Optionally, in this embodiment of the present invention, the first mapping table, the second mapping table, or the third mapping table may be jointly configured by the internal processing apparatus <NUM> in the switching device <NUM> and the host <NUM>. For example, the internal processing apparatus <NUM> may obtain, by means of device enumeration or in another manner, information about a physical device connected to the switching device <NUM> by using the M<NUM> PCIe downstream ports <NUM>, and configure the mapping table according to the obtained information, for example, store the information about the physical device in the mapping table. The host <NUM> may store, in the mapping table, configuration information, of the PCIe domain corresponding to the host <NUM>, obtained in a device enumeration (that is, discovering a physical device and/or a functional module under the host <NUM>) process or in another manner. However, this is not limited in this embodiment of the present invention.

In addition, the N<NUM> vEPs <NUM>, the mapping module <NUM>, and the M<NUM> mEPs <NUM> may be implemented by a register or by a register and a hardware circuit, and are simple in use without implementing a PCIe structure tree by using software code. In addition, a storage and forwarding process in a software implementation process is avoided by using a dual operation of configure write of a virtual device and a real device.

In this embodiment of the present invention, the PCIe system <NUM> may support multiple operation modes. The multiple operation modes may include at least one of the following modes: a VF direct mode, a VF shared mode, a PF shared mode, or an EP transparent transmission mode. The VF direct mode and the VF shared mode may be applied to an I/O device supporting SRIOV; the PF shared mode may be applied to an I/O device supporting multiple functions; the EP transparent transmission mode may be applied to a single-function I/O device.

Specifically, in the VF direct mode, the N<NUM> vEPs <NUM> are configured to virtualize virtual functions of the M<NUM> I/O devices <NUM>. For example, as shown in <FIG>, a PCIe system may include: two hosts: a host <NUM> and a host <NUM>; a PCIe switch; two endpoint devices: an EP <NUM> and an EP <NUM>; and a switching device. The switching device includes: two PCIe upstream ports respectively connected to the host <NUM> and the host <NUM>, two PCIe downstream ports respectively connected to the EP <NUM> and the EP <NUM> by using the PCIe switch, PCIe configuration spaces DP_CFG <NUM> and DP_CFG <NUM> respectively corresponding to the two PCIe downstream ports, an internal processing apparatus, a mapping module, a vEP <NUM>, and a vEP <NUM>. The vEP <NUM> is connected to one PCIe upstream port by using the DP_CFG <NUM>, and the vEP <NUM> is connected to the other PCIe upstream port by using the DP_CFG <NUM>. Each of the EP <NUM> and the EP <NUM> has a PF: a PF <NUM>, and a VF: a VF <NUM> corresponding to the PF <NUM>. In this case, the vEP <NUM> may be configured to virtualize the VF <NUM> of the EP <NUM>, so that the VF <NUM> is used by the host <NUM>. Specifically, the vEP <NUM> may be a combination of PCIe configuration spaces of the PF <NUM> and the VF <NUM> of the EP <NUM>. The vEP <NUM> may be configured to virtualize the VF <NUM> of the EP <NUM>, so that the VF <NUM> is used by the host <NUM>. Specifically, the vEP <NUM> may be a combination of PCIe configuration spaces of the PF <NUM> and the VF <NUM> of the EP <NUM>. However, this is not limited in this embodiment of the present invention.

Optionally, in the VF direct mode, PF drives of the M<NUM> I/O devices <NUM> may be loaded by the management BMC, and VF drives of the M<NUM> I/O devices <NUM> may be loaded by processors of the N<NUM> hosts <NUM>. Specifically, each host may load a VF drive, of an I/O device, virtualized by the vEP that is enumerated by the host. However, this is not limited in this embodiment of the present invention.

In the VF shared mode, the N<NUM> vEPs <NUM> are configured to virtualize physical functions and virtual functions of the M<NUM> I/O devices <NUM>. In this case, in the functions virtualized by the vEP <NUM>, only the VF may be used as a service port to be used by the host <NUM>, and the PF cannot be used as a service port to be used by the host <NUM>. Optionally, the vEP <NUM> may be specifically a PCIe configuration space of a virtualized PF. The PCIe configuration space may include an SRIOV capability space. However, this is not limited in this embodiment of the present invention. For example, as shown in <FIG>, the PCIe system may include: two hosts: a host <NUM> and a host <NUM>; a PCIe switch; an endpoint device: an EP <NUM>; and a switching device. The switching device includes: two PCIe upstream ports respectively connected to the host <NUM> and the host <NUM>, a PCIe downstream port connected to the EP <NUM> by using the PCIe switch, two PCIe configuration spaces DP_CFG <NUM> and DP_CFG <NUM> corresponding to the PCIe downstream port, an internal processing apparatus, a mapping module, a vEP <NUM>, and a vEP <NUM>. The EP <NUM> has a physical function (Physical Function, PF): a PF <NUM> and two virtual functions (Virtual Function, VF): a VF <NUM> and a VF <NUM> corresponding to the PF <NUM>. In this case, the vEP <NUM> and the vEP <NUM> may be specifically a PCIe configuration space, of the PF <NUM>, that carries an SRIOV capability space, so as to respectively virtualize the VF <NUM> and the VF <NUM>. Optionally, in the VF shared mode, the PF drives of the M<NUM> I/O devices <NUM> may be loaded by the management BMC. Different from the VF direct mode, the processors of the N<NUM> hosts <NUM> further need to load PF drives corresponding to the N<NUM> vEPs <NUM>, for enabling a VF virtualized by the N<NUM> vEPs <NUM>, so that the N<NUM> hosts <NUM> can normally use the VF.

In the PF shared mode, the N<NUM> vEPs <NUM> may be configured to virtualize physical functions of the M<NUM> I/O devices <NUM>. In this case, optionally, the M<NUM> I/O devices <NUM> may have only a PF and have no VF, that is, the PF in the M<NUM> I/O devices has no VF corresponding to the PF. Optionally, the vEP <NUM> may be specifically a PCIe configuration space of a PF virtualized by the vEP <NUM>, and the PCIe configuration space may not include an SRIOV capability space. For example, as shown in <FIG>, the PCIe system may include: two hosts: a host <NUM> and a host <NUM>; a PCIe switch; an endpoint device: an EP <NUM>; and a switching device. The switching device includes: two PCIe upstream ports respectively connected to the host <NUM> and the host <NUM>, a PCIe downstream port connected to the EP <NUM> by using the PCIe switch, two PCIe configuration spaces DP_CFG <NUM> and DP_CFG <NUM> corresponding to the PCIe downstream port, an internal processing apparatus, a mapping module, a vEP <NUM>, and a vEP <NUM>. The EP <NUM> has two PFs: a PF <NUM> and a PF <NUM>, and has no VF. In this case, the vEP <NUM> may be configured to virtualize the PF <NUM>, so that the PF <NUM> is used by the host <NUM>. The vEP <NUM> may be configured to virtualize the PF <NUM>, so that the PF <NUM> is used by the host <NUM>. However, this is not limited in this embodiment of the present invention. Optionally, in the PF shared mode, the PF drives of the M<NUM> I/O devices <NUM> may be loaded by the processors of the N<NUM> hosts <NUM>. Specifically, a processor of each host <NUM> may load a drive of a PF virtualized by a vEP enumerated by the processor.

In the EP transparent transmission mode, the switching device <NUM> may transparently transmit information transmitted between the M<NUM> I/O devices <NUM> and the N<NUM> hosts <NUM> that are connected to the switching device <NUM>. In this case, the M<NUM> I/O devices <NUM> may be a single-function device, that is, have only a PF and have no VF. For example, as shown in <FIG>, a PCIe system may include: two hosts: a host <NUM> and a host <NUM>; a PCIe switch; three endpoint devices: an EP <NUM>, an EP <NUM>, and an EP <NUM>; and a switching device. The switching device includes: two PCIe upstream ports respectively connected to the host <NUM> and the host <NUM>, a PCIe downstream port connected to the EP <NUM> and the EP <NUM> by using the PCIe switch, and a PCIe downstream port directly connected to the EP <NUM>. Each of the EP <NUM>, the EP <NUM>, and the EP <NUM> has only one PF: a PF <NUM>. The switching device may not store a PCIe configuration space of any PCIe downstream port. In this case, the host <NUM> may use PFs of the EP <NUM> and the EP <NUM>. The host <NUM> may use a PF of the EP <NUM>. All modules in the switching device perform transparent transmission, and cannot implement a function of sharing a same EP by multiple hosts. Optionally, in the EP transparent transmission mode, a VF drive and a PF drive do not need to be loaded.

Optionally, the PCIe system <NUM> may always operate in a particular one of the foregoing multiple operation modes, or may be switched between the foregoing multiple operation modes. Optionally, the PCIe system <NUM> may further support another operation mode. This is not limited in this embodiment of the present invention.

It should be understood that examples of <FIG> are intended to help a person skilled in the art better understand this embodiment of the present invention, but not to limit the scope of this embodiment of the present invention. Apparently, a person skilled in the art may perform various equivalent modifications or changes according to the provided examples of <FIG>. The modifications or the changes also fall into the scope of this embodiment of the present invention.

It should be further understood that the examples are all described by using an example in which the PCIe system includes one switching device. Optionally, the PCIe system may include multiple switching devices. As shown in <FIG>, the PCIe system <NUM> includes two switching devices <NUM>: a switching device <NUM> and a switching device <NUM>. Each switching device may be connected to two hosts and two I/O devices. Specifically, the switching device <NUM> may be separately connected to a host <NUM>, a host <NUM>, an IO <NUM>, and an IO <NUM>. The switching device <NUM> is separately connected to a host <NUM>, a host <NUM>, an IO <NUM>, and an IO <NUM>. In this way, if a switching device has an exception, for example, the switching device <NUM> connected to the IO <NUM> and the IO <NUM> has an exception, exception processing, for example, a reset operation or related service interruption, may be performed according to an actual situation. If the switching device <NUM> connected to the IO <NUM> and the IO <NUM> normally runs, the host <NUM> and the host <NUM> can still run normally, thereby enhancing reliability and stability of the system.

It should be further understood that the foregoing embodiments are all described by using an example in which the switching device <NUM> is specifically one physical device. Optionally, the switching device <NUM> may be specifically multiple physical devices. For example, as shown in <FIG> and <FIG>, the switching device <NUM> may be in an AB-chip form, that is, the switching device <NUM> may be specifically a host switching device 220a and an I/O switching device 220b. The host switching device 220a and the I/O switching device 220b may be connected by using an Ethernet interface or a switched interface of another type. In an optional embodiment, the host switching device 220a may include: N<NUM> PCIe upstream ports <NUM> and at least one first switched interface that is configured to connect to the I/O switching device 220b. The I/O switching device 220b may include: at least one second switched interface configured to connect to the host switching device 220a, an internal processing apparatus <NUM>, and M<NUM> PCIe downstream ports <NUM>. Optionally, as shown in <FIG>, the at least one first switched interface and the at least one second switched interface may be directly connected. Alternatively, as shown in <FIG>, the at least one first switched interface and the at least one second switched interface may be connected by using an Ethernet switch or an internal switching device of another type. The internal switching device may provide multiple switched interfaces configured to connect the host switching device and the I/O switching device. In this case, the at least one first switched interface may be configured to connect to the internal switching device, and the at least one second switched interface may be configured to connect to the internal switching device. This is not limited in this embodiment of the present invention.

In another optional embodiment, the internal processing apparatus <NUM> may include a processing apparatus (that is, a first processing apparatus) located in the host switching device 220a and a processing apparatus (that is, a second processing apparatus) located in the I/O switching device 220b. In this case, the first internal processing apparatus may be separately connected to the multiple PCIe upstream ports and the at least one first switched interface by using an internal connection line of the host switching device. The second internal processing apparatus may be separately connected to the at least one PCIe downstream port and the at least one second switched interface by using an internal connection line of the I/O switching device. However, this is not limited in this embodiment of the present invention.

In this case, optionally, the first internal processing apparatus is configured to: receive, by using the multiple PCIe upstream ports, a first data packet from the at least one host; process the first data packet to obtain a first data packet on which first processing is performed; and transmit, to the internal switching device by using the at least one first switched interface, the first data packet on which first processing is performed.

The second internal processing apparatus is configured to: receive, by using the at least one second switched interface, the first data packet on which first processing is performed that is transmitted by the internal switching device; process the first data packet on which first processing is performed to obtain a first data packet on which second processing is performed; and transmit, to the at least one I/O device by using the at least one PCIe downstream port, the first data packet on which second processing is performed.

Optionally, the second internal processing apparatus is further configured to: receive, by using the at least one PCIe downstream port, a second data packet from the at least one I/O device; process the second data packet to obtain a second data packet on which first processing is performed; and transmit, to the internal switching device by using the at least one second switched interface, the second data packet on which first processing is performed.

The first internal processing apparatus is further configured to: receive, by using the at least one first switched interface, the second data packet on which first processing is performed that is transmitted by the internal switching device; process the second data packet on which first processing is performed to obtain a second data packet on which second processing is performed; and transmit, to the at least one host by using the multiple PCIe upstream ports, the second data packet on which second processing is performed.

Optionally, the I/O switching device 220b may further include N<NUM> vEPs <NUM>, a mapping module <NUM>, and M<NUM> mEPs <NUM>. The host switching device 220a may further include N<NUM> pieces of DP_CFG <NUM> and N<NUM> vEPs <NUM>.

As shown in <FIG>, a switching device <NUM> may include a host switching device <NUM> and an I/O switching device <NUM>, and a switching device <NUM> may include a host switching device <NUM> and an I/O switching device <NUM>. The host switching devices and the I/O switching devices are connected by using an Ethernet interface (for example, a Media Access Control (Media Access Control, MAC) interface). In this case, if the host switching device <NUM> connected to a host <NUM> and a host <NUM> has an exception, the host switching device <NUM> may be replaced according to an actual situation, but a host <NUM> and a host <NUM> can still continue accessing an I/O device, and a service can continue being operated. If the I/O switching device <NUM> connected to an IO <NUM> and an IO <NUM> has an exception, because the I/O switching device <NUM> connected to an IO <NUM> and an IO <NUM> is normal, not only the host <NUM> and the host <NUM> can use the IO <NUM> and the IO <NUM> by using the host switching device <NUM>, an Ethernet switch, and the I/O switching device <NUM>, but also the host <NUM> and the host <NUM> can use the IO <NUM> and the IO <NUM> by using the host switching device <NUM>, the Ethernet switch, and the I/O switching device <NUM>, so that the four hosts can all continue operating the service, thereby further enhancing the reliability and the stability of the system.

Specifically, in this embodiment of the present invention, the host switching device may include the multiple PCIe upstream ports configured to connect to the at least one host. The I/O switching device may include the internal processing apparatus, and the at least one PCIe downstream port configured to connect to the at least one I/O device.

In another optional embodiment, if the switching device includes at least one mEP, a mapping module, and at least one vEP, the at least one mEP, the mapping module, and the at least one vEP may be deployed only in the I/O switching device. Alternatively, as shown in <FIG>, the host switching device may include at least one vEP, and the at least one vEP in the host switching device is the same as at least one vEP in the I/O switching device in a one-to-one correspondence. However, this is not limited in this embodiment of the present invention.

It should be understood that "the host switching device" and "the I/O switching device" are merely names for distinguishing different switching devices. Alternatively, "the host switching device" may be referred to as a first switching device, and "the I/O switching device" may be referred to as a second switching device. The names should not constitute any limitation on the protection scope of the embodiments of the present invention.

Therefore, the PCIe system provided in this embodiment of the present invention includes the at least one host, the switching device, and the at least one I/O device. The switching device includes the multiple PCIe upstream ports configured to connect to the at least one host, the at least one PCIe downstream port configured to connect to the at least one I/O device, and the internal processing apparatus. The processing apparatus is connected to the at least one PCIe downstream port by using the internal connection line, and is configured to: transmit a configuration read/write packet to the at least one PCIe downstream port by using the internal connection line, receive a configuration read/write response packet transmitted by the at least one PCIe downstream port by using the internal connection line, and determine, according to a completer identification carried in the configuration read/write response packet, that the switching device is connected to an I/O device whose identification is the completer identification. In this way, when a switching device in the PCIe system is faulty, only at least one host and at least one I/O device that are connected to the switching device are affected, and another switching device, and a host and an I/O device that are connected to the another switching device in the system are not affected. Therefore, compared with an external management CPU in the prior art, stability and reliability of the system can be improved.

In addition, a conventional PCIe technology in the prior art is mainly applied to a board, and has relatively many limitations, for example, problems such as a limited quantity of PCIe ports of a host, limited space, and poor expansion. A switching device is disposed as two physical devices, and a PCIe domain is borne on the Ethernet for transmission. In this way, limitations of space and distance can be broken through, a PCIe topology can be flexibly set up, an I/O device can be flexibly deployed, and a problem in the prior art that the PCIe domain cannot be remotely deployed is resolved.

An embodiment of the present invention further provides a switching device. The switching device may be as that described in the foregoing system embodiment. For concision, details are not described herein again.

The PCIe system and the switching device provided in the embodiments of the present invention are described in detail above with reference to <FIG>. An operation procedure of the PCIe system is described in detail below.

<FIG> shows a method <NUM> for initializing a PCIe system according to an embodiment of the present invention. The initialization method <NUM> may be applied to the PCIe system in the foregoing embodiment.

S310: The management BMC sends an initialization instruction to the internal processing apparatus in the switching device.

After being powered on, the management BMC may send the initialization instruction to the processing apparatus. The initialization instruction may be used to instruct the processing apparatus to perform an operation, for example, perform device enumeration and/or perform initialization configuration on a module (or a component) in the switching device, related to system initialization.

S320: After receiving the initialization instruction, the processing apparatus of the switching device enumerates, according to the initialization instruction, a physical device connected to the switching device by using the at least one PCIe downstream port of the switching device, where the physical device includes the at least one I/O device.

In this embodiment of the present invention, the management BMC may allocate an identification, for example, allocate a device identifier, to the internal processing apparatus in the switching device. The device identifier may include three fields: a bus number, a device number, and a function number. The field of the bus number field indicates a number of a bus on which the device is located, the field of the device number indicates a number of the device, and the field of the function number indicates a number of a function possessed by the device. Usually, the device identifier may also be referred to as a bus device function (Bus Device Function, BDF) number. Optionally, the device identifier allocated by the management BMC to the processing apparatus may be unique in the system. For example, in an X86 system, the management BMC may allocate <NUM>. <NUM> as the device identifier of the processing apparatus. However, this is not limited in this embodiment of the present invention.

When performing device enumeration, the processing apparatus may specifically detect a real device connected to the switching device by using the at least one PCIe downstream port. Specifically, the internal processing apparatus may send multiple TLPs to the at least one PCIe downstream port by using the internal connection bus. The TLP may be specifically a configuration read/write request, a value of a RID carried in the TLP may be set to a BDF of the processing apparatus, and a CID may be numbered in sequence. When receiving the configuration read/write packet, a PCIe downstream port may forward the received configuration read/write request to a physical device connected to the PCIe downstream port. The physical device may send, according to the received configuration read/write request, a configuration read/write response packet, for example, a CPL packet or a CPLD packet, to the PCIe downstream port connected to the physical device. The configuration read/write response packet may carry the RID and the CID that are in the received configuration read/write packet. In this way, when receiving a configuration read/write response packet by using a PCIe downstream port, the processing apparatus may determine that the switching device is connected to a physical device whose BDF number is a CID in the configuration read/write response packet, and may further obtain information that is about the physical device and that is carried in the configuration read/write response packet. However, this is not limited in this embodiment of the present invention.

S330: The switching device sends, to the management BMC, a device enumeration result of the internal processing apparatus.

Specifically, the internal processing apparatus may report the device enumeration result to the management BMC by using the multiple PCIe upstream ports. The enumeration result may include the physical device connected to the switching device by using the at least one PCIe downstream port and a topology structure of the physical device, or may include a PCIe structure tree. This is not limited in this embodiment of the present invention.

S340: Each of the at least one host enumerates a device connected to each host.

A processor of each of the at least one host may enumerate a device connected to the processor. The device may include a virtual device (that is, a functional module) and/or a physical device, for example, a PCIe upstream port that is of the switching device and that is connected to the host, and at least one vEP. The enumeration may end at the vEP. However, this is not limited in this embodiment of the present invention.

Specifically, a processor of a host may send a configuration read packet, where the configuration read packet ends at a vEP of the switching device. After receiving the configuration read packet, the switching device may return a configuration read response packet to the host by using a PCIe upstream port. After receiving the configuration read response packet, the processor of the host may send a configuration write packet to the vEP. Specifically, the configuration write packet may be used to access a register of the vEP. Optionally, each configuration write packet may access space of <NUM> KB of the register each time. The configuration write packet still ends at the vEP of the switching device. After receiving the configuration write packet, the vEP of the switching device may forward, by using a PCIe downstream port, the configuration write packet to an I/O device corresponding to the vEP (that is, the vEP is configured to virtualize a function of the I/O device). After receiving the configuration write packet, the I/O device corresponding to the vEP may return a configuration write response packet to the switching device by using the PCIe downstream port, where the configuration write response packet may carry a BDF number of the I/O device. Optionally, after receiving the configuration write response packet, the mapping module of the switching device may search a second mapping table according to a CID carried in the configuration write response packet, to obtain a BDF number of the vEP corresponding to the configuration write response packet, and may replace the CID in the configuration write response packet with the BDF number of the vEP corresponding to the CID. Then, the mapping module of the switching device may determine, according to a RID carried in the configuration write response packet, to send the configuration write response packet to a PCIe upstream port of the switching device, and return the configuration write response packet to the host by using the PCIe upstream port.

Optionally, the at least one host and/or the switching device may write, to the vEP and/or the mEP by means of an auto-learning mechanism, information (for example, the BDF number) obtained in the foregoing process of performing enumeration. This is not limited in this embodiment of the present invention.

S350: The at least one host loads a VF drive and a PF drive of the at least one I/O device. Specifically, the VF drive may be loaded by a processor of the host, and the PF drive may be loaded by the processor of the host or the management BMC according to different operation modes. Specifically, in a VF direct mode and a VF shared mode, the PF drive of the I/O device may be loaded by the management BMC. In addition, in the VF shared mode, the processor of the host may further load a PF drive of the vEP, that is, a processor of each host may load a PF drive of a vEP enumerated by the processor of the host. In a PF shared mode, the PF drive of the I/O device may be loaded by the processor of the host, that is, a processor of each host may load a PF drive (or a PF drive of a PF virtualized by a vEP) of an I/O device corresponding to a vEP enumerated by the processor of the host. However, this is not limited in this embodiment of the present invention.

In another optional embodiment, after S330, the method <NUM> may further include the following step.

The internal processing apparatus in the switching device configures a functional module in the switching device. Specifically, the functional module includes: the at least one vEP, the mapping module, and the at least one mEP. In this case, the processing apparatus may establish a mapping relationship between the at least one mEP and the at least one vEP, that is, establish a mapping relationship between a function of the at least one I/O device and a function that is virtualized by the vEP and that can be used by the at least one host. The processing apparatus may specifically store, in the functional module of the switching device, information about a physical device under the switching device. For example, the processing apparatus may configure the first mapping table, the second mapping table, and the third mapping table in the foregoing embodiment according to the obtained device information. The information about the physical device may be obtained in a process of performing device enumeration by the processing apparatus. For example, information, for example, a BDF number, about each physical device is obtained from a configuration read/write response packet. Alternatively, the information about the physical device may be delivered by the management BMC. Correspondingly, optionally, before the processing apparatus configures the functional module in the switching device, the method <NUM> may further include: The management BMC sends configuration management information to the processing apparatus. The configuration management information may include information, for example, a currently used operation mode (for example, any one of the foregoing operation modes), configuration information (for example, a quantity of PCIe configuration spaces corresponding to the PCIe downstream ports, a quantity of vEPs under each PCIe configuration space, a connection relationship between the PCIe downstream port and the I/O device, or a MAC address of each PCIe port) of the PCIe downstream port and the PCIe upstream port, or information about a PCIe configuration space of the I/O device, needed for configuring the functional module of the switching device. However, this is not limited in this embodiment of the present invention.

After performing the initialization procedure, the system may operate a service. In this case, for a configuration write packet in a configuration packet sent by the host, the switching device (for example, the mapping module of the switching device) needs to modify a CID in the configuration write packet. For a Mem read/write packet delivered by the host, an address of the Mem read/write packet needs to be modified (for example, base address replacement is performed on addresses corresponding to different domains, and an offset is unchanged). For a completion packet CPLD/CPL sent by the host, only a CID needs to be modified. For a Mem read/write packet sent by the I/O device, a RID of the Mem read/write packet may be modified, and an address does not need to be modified. For a completion packet CPLD/CPL sent by the I/O device, only a CID of the completion packet CPLD/CPL needs to be modified. However, this is not limited in this embodiment of the present invention.

It should be understood that, for the initialization method, reference may be made to the specific description of the foregoing system embodiment. For concision, details are not described herein again. In addition, the description of each embodiment of this application emphasizes differences between the embodiment and another embodiment, and similarities between the embodiment and the another embodiment may be used for mutual reference.

It should be understood that sequence numbers of the foregoing processes do not mean a performing order. The performing order of the processes should be determined according to functions and internal logic of the processes, and should not constitute any limitation on an implementation process of this embodiment of the present invention.

In addition, an embodiment of the present invention further provides a switching system. The switching system includes a first switching device and a second switching device. The first switching device and the second switching device are connected by using a network.

The first switching device may include: multiple PCIe upstream ports, configured to connect to at least one host.

The second switching device may include: at least one PCIe downstream port, configured to connect to at least one device.

Optionally, the switching system may further include a network switching device located in the network. The network switching device may have multiple switched interfaces, configured to connect the first switching device and the second switching device.

In this case, optionally, the first switching device further includes: at least one first switched interface, configured to connect to the network switching device. The second switching device further includes: at least one second switched interface, configured to connect to the network switching device. Optionally, the network switching device may be specifically an Ethernet switch shown in <FIG>. However, this is not limited in this embodiment of the present invention.

Optionally, the first switching device may be specifically the host switching device, and the second switching device may be specifically the I/O switching device. As shown in <FIG> and <FIG>, the host switching device 220a and the I/O switching device 220b may be connected by using an Ethernet interface or a switched interface of another type. In an optional embodiment, the host switching device 220a may include N<NUM> PCIe upstream ports <NUM> and at least one first switched interface that is configured to connect to the I/O switching device 220b. The I/O switching device 220b may include: at least one second switched interface configured to connect to the host switching device 220a, an internal processing apparatus <NUM>, and M<NUM> PCIe downstream ports <NUM>.

Optionally, as shown in <FIG>, the at least one first switched interface and the at least one second switched interface may be directly connected. Alternatively, as shown in <FIG>, the at least one first switched interface and the at least one second switched interface may be connected by using an Ethernet switch or a network switching device of another type. The network switching device may provide multiple switched interfaces configured to connect the host switching device and the I/O switching device. In this case, the at least one first switched interface may be configured to connect to the network switching device, and the at least one second switched interface may be configured to connect to the network switching device. This is not limited in this embodiment of the present invention.

In an optional embodiment, the second switching device may be configured to: receive a first data packet transmitted by the first switching device by using the network, process the first data packet to obtain a second data packet complying with a PCIe protocol, and transmit the second data packet to a target I/O device of the second data packet.

Optionally, the second switching device may be configured to convert the first data packet to the second data packet complying with the PCIe protocol. Optionally, the conversion may include parsing processing, packet format conversion processing, time sequence conversion processing, and the like.

However, this is not limited in this embodiment of the present invention.

Optionally, the first data packet may be specifically a data packet complying with a network protocol of the network. In an optional embodiment, as shown in <FIG>, the network may be the Ethernet. In this case, the second switching device may be specifically configured to convert the first data packet complying with an Ethernet protocol to the second data packet complying with the PCIe protocol. However, this is not limited in this embodiment of the present invention.

Optionally, the second switching device may be further configured to: determine, according to information carried in a received data packet or according to a port configured to receive the data packet, a target device of the data packet, and send the data packet according to the target device of the data packet. For example, the second switching device may further determine, according to information carried in the first data packet, a target I/O device of the first data packet, and send, to the target I/O device of the first data packet, the second data packet that is obtained by processing the first data packet. However, this is not limited in this embodiment of the present invention.

Optionally, the first switching device may receive, by using the at least one first switched interface, an uplink data packet transmitted by the network switching device, and may transmit a downlink data packet to the network switching device by using the at least one first switched interface. For example, the first switching device may transmit the first data packet to the second switching device by using the at least one switched interface. However, this is not limited in this embodiment of the present invention. Optionally, the second switching device may receive, by using the at least one second switched interface, a downlink data packet transmitted by the network switching device, and may transmit an uplink data packet to the network switching device by using the at least one second switched interface. For example, the second switching device may receive, by using the at least two switched interfaces, the first data packet transmitted by the first switching device. However, this is not limited in this embodiment of the present invention.

Optionally, the second switching device further includes: a first internal processing apparatus, connected to the at least one PCIe downstream port by using an internal connection line of the second switching device.

Optionally, that the second switching device is configured to process the first data packet to obtain a second data packet complying with a PCIe protocol includes: the first internal processing apparatus is configured to process the first data packet, to obtain the second data packet complying with the PCIe protocol.

Optionally, the second switching device may be further configured to process an uplink data packet.

Optionally, a first processing apparatus in the second switching device is configured to: receive a third data packet from the at least one I/O device, process the third data packet to obtain a fourth data packet complying with a network protocol of the network, and transmit the fourth data packet to the first switching device by using the network.

Optionally, the first processing apparatus may perform conversion processing on a third data packet complying with the PCIe protocol, to obtain a fourth data packet complying with the network protocol of the network.

Optionally, the first switching device may process a received uplink data packet and/or downlink data packet.

Optionally, the first switching device further includes: a second internal processing apparatus, connected to the multiple PCIe upstream ports by using an internal connection line of the first switching device.

Optionally, the second internal processing apparatus may be configured to process a received uplink/downlink data packet.

In an optional embodiment, the second internal processing apparatus is configured to: receive a fifth data packet from the at least one host, process the fifth data packet to obtain the first data packet complying with the network protocol of the network, and transmit the first data packet to the second switching device by using the network.

Optionally, the fifth data packet may comply with the PCIe protocol. The second processing apparatus may be specifically configured to convert the fifth data packet to the first data packet, where the first data packet complies with the network protocol of the network.

Optionally, the second processing apparatus may further determine, according to information carried in the fifth data packet and/or a PCIe upstream port receiving the fifth data packet, a first switched interface corresponding to the fifth data packet, and transmit the first data packet to the second switching device by using the corresponding first switched interface. However, this is not limited in this embodiment of the present invention.

In another optional embodiment, the second internal processing apparatus is configured to: receive, by using the network, the fourth data packet transmitted by the first switching device, process the fourth data packet to obtain a sixth data packet complying with the PCIe protocol, and transmit the sixth data packet to a target host of the sixth data packet.

Optionally, the second processing apparatus may receive, by using the network, a fourth data packet transmitted by the second switching device, where the fourth data packet may be a data packet complying with a network protocol of the network. The second processing apparatus may perform conversion processing on the fourth data packet to obtain the sixth data packet complying with the PCIe protocol.

Optionally, the second processing apparatus may be configured to: determine, according to information carried in a received data packet or according to a port configured to receive the data packet, a target device of the data packet, and send the data packet according to the target device of the data packet. For example, the second processing apparatus may determine, according to information carried in the fourth data packet, a target host of the fourth data packet, and transmit, to the target host of the fourth data packet, the sixth data packet that is obtained by processing the fourth data packet. However, this is not limited in this embodiment of the present invention.

Optionally, the first switching device further includes: at least one virtual endpoint device module, connected to the multiple PCIe upstream ports, and configured to virtualize a function of the at least one I/O device, so that the function is used by the at least one host.

The second switching device further includes:.

Optionally, none of the at least one PCIe downstream port has any PCIe configuration space, and each of the multiple PCIe upstream ports has a PCIe configuration space.

Optionally, the first switching device further includes at least one PCIe configuration space corresponding to the at least one PCIe downstream port. The at least one virtual endpoint device module is specifically connected to the multiple PCIe upstream ports by using the at least one PCIe configuration space corresponding to the at least one PCIe downstream port.

Optionally, the at least one PCIe configuration space corresponding to the at least one PCIe downstream port may be specifically the N<NUM> pieces of DP_CFG <NUM>. However, this is not limited in this embodiment of the present invention. Optionally, each of the at least one PCIe downstream port may correspond to zero, one, or more PCIe configuration spaces. This is not limited in this embodiment of the present invention.

Optionally, a first mirroring endpoint device module in the at least one mirroring endpoint device module is specifically a third mapping table, and the third mapping table is used to store a base address register BAR address and a BAR size of a virtual function of a first I/O device in the at least one I/O device, where the first mirroring endpoint device module is configured to store PCIe configuration content of the first I/O device.

Optionally, a first virtual endpoint device module in the at least one virtual endpoint device module is specifically configured to virtualize a physical function of a second I/O device in the at least one I/O device, so that the physical function is used by a first host in the at least one host, where a physical function drive of the second I/O device is loaded by a processor of the first host.

Optionally, a second virtual endpoint device module in the at least one virtual endpoint device module is specifically configured to virtualize a virtual function of a third I/O device in the at least one I/O device, so that the virtual function is used by a second host in the at least one host, where a physical function drive of the third I/O device is loaded by a management baseboard management controller BMC in the at least one host.

Optionally, a third virtual endpoint device module in the at least one virtual endpoint device module is specifically configured to virtualize a physical function and a virtual function of a fourth I/O device in the at least one I/O device, so that the physical function and the virtual function are used by a third host in the at least one host, where a physical function drive of the fourth I/O device is loaded by the management BMC in the at least one host, and a physical function drive of the third virtual endpoint device module is loaded by a processor of the third host.

It should be understood that the at least one virtual endpoint device module may be specifically the N<NUM> vEPs <NUM>, the at least one mirroring endpoint device module may be specifically the M<NUM> mEPs <NUM>, and the mapping module may be specifically the mapping module <NUM>. Therefore, for specific implementations of the at least one virtual endpoint device module, the at least one mirroring endpoint device module, and the mapping module, refer to the foregoing description. For concision, details are not described herein again.

Optionally, the first switching device may be specifically the host switching device, and the second switching device may be specifically the I/O switching device. For specific implementations of the first switching device and the second switching device, refer to the foregoing description. For concision, details are not described herein again. It should be understood that "the host switching device", "the I/O switching device", "the first switching device", and "the second switching device" herein are merely names for distinguishing different switching devices. The names should not constitute any limitation on the protection scope of the embodiments of the present invention.

An embodiment of the present invention further provides a PCIe system, including: at least one host, the switching system in the foregoing embodiment, and at least one I/O device.

It should be understood that in the embodiments of the present invention, the term "multiple" may indicate at least two or three, and the term "and/or" describes only an association relationship for describing associated objects, and represents that three relationships may exist.

A person of ordinary skill in the art may be aware that, in combination with the examples described in the embodiments disclosed in this specification, method steps and units may be implemented by electronic hardware, computer software, or a combination thereof. To clearly describe the interchangeability between the hardware and the software, the foregoing has generally described steps and compositions of each embodiment according to functions. A person of ordinary skill in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.

It may be clearly understood by a person skilled in the art that, for the purpose of convenient and brief description, for detailed working processes of the foregoing system, apparatus, and unit, reference may be made to corresponding processes in the foregoing method embodiments, and details are not described herein again.

For example, multiple units or components may be combined or may be integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces, indirect couplings or communication connections between the apparatuses or units, or electrical connections, mechanical connections, or connections in other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Apart or all of the units may be selected according to actual needs to achieve the objectives of the solutions in the embodiments of the present invention.

Claim 1:
A switching system, comprising a first switching device and a second switching device, wherein the first switching device and the second switching device are connected by using a network;
the first switching device comprises:
multiple Peripheral Component Interconnect Express, PCIe, upstream ports (<NUM>), configured to connect to at least one host (<NUM>);
the second switching device comprises:
at least one PCIe downstream port (<NUM>), configured to connect to at least one input/output, I/O, device (<NUM>); and
the second switching device is configured to:
receive a first data packet transmitted by the first switching device by using the network, convert the first data packet to a second data packet complying with a PCIe protocol, and transmit the second data packet to a target I/O device of the second data packet;
wherein the second switching device further comprises: a first internal processing apparatus (<NUM>), connected to the at least one PCIe downstream port by using an internal connection line of the second switching device, wherein the first internal processing apparatus is configured to determine at least one I/O device connected to the second switching device by using the at least one PCIe downstream port and obtain information of the at least one I/O device connected to the second switching device, the information comprising at least one of a Bus Device Function, BDF, and a topology structure; and
the first internal processing apparatus is further configured to convert the first data packet to the second data packet complying with the PCIe protocol.