Peripheral devices integrated into a processing chain

Provided in some embodiment is a computer system, including a first peripheral device, having a first external data input, a first peripheral storage device to store the measurement data, a first peripheral device output to couple to a system interconnect of the computer system. The first peripheral device capable of receiving measurement data via the external data input the first peripheral device capable of transferring at least a portion of the measurement data to a second peripheral device of the computer system via the system interconnect, and where the second peripheral device is capable of processing at least a portion of the measurement data transferred to the second peripheral device.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a system and method of offloading processing functions between devices of a computer system and more particularly to the exchange of data from one peripheral device to another peripheral device for processing.

2. Description of the Related Art

Data acquisition (DAQ) typically includes acquiring and processing of data in a computer system. Some DAQ applications may involve in-line, real time, processing of data. There may be several motivations for the in-line processing. Among these motivations may be real time and deterministic demands of the application itself. It is not uncommon, however, for some computer based DAQ systems to suffer from performance issues. Some of the factors limiting performance may include the overhead of the operating system (OS) and other applications running on the host processor, the lack of deterministic execution in a non-real time OS, the lack of computer power for the specified application and computer power that does not scale with the demands of the application etc.

Several solutions have been implemented to try to get around these issues. In some cases, at least a portion of processing is done on a DAQ device (e.g., a DAQ board), or similar computer device. In some situations the processing load is offloaded from a device to a host processor or from a host processor to a device so that processing may be accomplished in a timely and deterministic manner. Some of these solutions may involve some form of a dedicated co-processor that may offload computational tasks from a host processor in a way that helps to make the application more deterministic and efficient. These solutions, however, are generally limited to transfer of data between a device and the host processor, and typically rely on fixed connections between the co-processor and I/O channels that do not provide a flexible solution. Accordingly it is desirable to provide a system and method for the transfer and processing of data between devices that may provide a flexible/customizable solution.

SUMMARY

Described herein are embodiments relating to a system and method of transferring data from one peripheral device to another for processing. For example, in one embodiment, provided is a computer system, including a first peripheral device, having a first external data input, a first peripheral storage device to store the measurement data, and a first peripheral device output to couple to a system interconnect of the computer system. The first peripheral device is capable of receiving measurement data via the external data input. The first peripheral device is capable of transferring at least a portion of the measurement data to a second peripheral device of the computer system via the system interconnect and the second peripheral device is capable of processing at least a portion of the measurement data transferred to the second peripheral device.

In another embodiment, provided is a computer system that includes a system interconnect, a first peripheral device to receive measurement data via an external input and a second peripheral device that includes a processor to process measurement data. The first peripheral device has an output coupled to the system interconnect and the second peripheral device has an input coupled to the system interconnect. The first peripheral device is able to directly transfer to the second peripheral device, via the system interconnect, at least a portion of the measurement data received by the first peripheral device via the external input. The processor of the second peripheral device is able to process at least a portion of the measurement data transferred to the second peripheral device.

In yet another embodiment, provided is a method that includes providing measurement data at an external input of a first peripheral device of a computer system and transferring measurement data directly from the first peripheral device to a second peripheral device via a system interconnect. The second peripheral device includes a processor able to process at least a portion of the measurement data transferred. The method also includes directly transferring measurement data via the system interconnect that includes transferring measurement data from the first peripheral device to the second peripheral device without passing the data through a processor or memory of a host central processing unit of the computer system.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present invention as defined by the appended claims. It is noted that the word “may” is used throughout this application in a permissive sense (e.g., having the potential to, being able to), not a mandatory sense (e.g., must).

DETAILED DESCRIPTION

Definitions

FIFO—a first in first out device. A FIFO is commonly constructed from a static random access memory (SRAM). Data are output from a FIFO in the sequence it arrived, that is, the first arriving data item is the first to be sent.

Streaming—refers to the quasi-continuous transfer of data. Data may be streamed from a source device (e.g., a data acquisition device) to a target device (e.g., an output device) across an interconnect medium that may include combinations of networks, buses, switched fabrics and point-to-point links. Data may be streamed through intermediate devices and may be modified by those devices as it is streamed. Streaming is generally associated with large quantities of data being transferred in real-time.

Peripheral Device—refers to a device coupled to a computer system (e.g., one or more modular boards), that is not a host device (e.g., host processor, such as a CPU and/or host memory). A peripheral device may include a printed circuit board (PCB) or card that is installed into a modular computer system via an expansion slot (e.g., a slot of a rack or chassis). A peripheral device may include supporting circuitry such as processors, bus interface circuitry, storage elements (such as random access memory (RAM) and read-only memory (ROM)), a peripheral controller/processor and its associated firmware.

Measurement Data—refers to data received at an external input of a computer system device. For example, measurement data may refer to signals/data received at an external input to a data acquisition board, a computer based-instrument, or a similar test and measurement device. Measurement data may include a signal (e.g., an analog of digital signal) and/or data associated with a signal.

In the following description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, one having ordinary skill in the art should recognize that the invention may be practiced without these specific details. In some instances, well-known circuits, structures, and techniques have not been shown in detail to avoid obscuring the present invention.

As discussed in more detail below, certain embodiments include a computer system having multiple peripheral devices that may be configured to share data and operate together to perform one or more processing functions. In some embodiments, data on one peripheral device may be transferred to another peripheral device for processing by the other peripheral device. In some embodiments, the transfer of data may occur directly between peripheral devices without passing data to or from a host processor. In some embodiments, multiple peripheral devices may be configured to input, transfer, process and/or output data independent of a central processing unit of a host.

In certain embodiments, peripheral devices may be configured to perform certain processing functions (e.g., process monitoring, equipment testing, system control, robotic control etc.) In some embodiments, the functionality of the system may be augmented by the addition of an expansion board (and associated peripheral device). Peripheral devices may include input devices, output devices, input/output devices, processors, controllers, communications devices, reconfigurable devices, customized hardware devices, storage devices and so and so forth. In some embodiments, the processing capability of a system may be increased by the addition of a new or replacement processor on peripheral devices. In some embodiments, peripheral device configuration may be performed automatically by software running on the computer system and it may involve human input. Embodiments may display (graphically or otherwise) system information and a system user may utilize that displayed information to select devices, boards, data flow among devices and many other system characteristics. Embodiments may be configured to perform multiple tasks concurrently. In some embodiments, device configuration may involve sending configuration information to a peripheral device, where the information may either directly configure the peripheral device or it may be used to configure the peripheral device.

In some embodiments, system configuration may involve configuring peripheral devices for sharing data (e.g., sending or receiving data) with other devices (notably other peripheral devices). In certain embodiments, data may be shared (and processed) as follows. A peripheral device may receive data from a single source (e.g., a peripheral input device), process the received data and then send processed data to a single target (e.g., a peripheral output device). In some embodiments, such peripheral-to-peripheral data transfers may be performed without routing data through a host CPU or host memory. Peripheral-to-peripheral transfers like this may be referred to as “direct” peripheral-to-peripheral transfers. In certain embodiments, data may be sent between peripheral devices using a host controller, which may read data from a source peripheral device and write data to a target peripheral device. Peripheral-to-peripheral transfer in which data is routed through a host controller's CPU or a host controller's memory may be referred to as “indirect” transfers.

In some embodiments, a single peripheral device may send data to a number of target devices (e.g., target peripheral devices). In such embodiments, different portions of output data may be sent to different targets devices so that, for example, a large task may be subdivided among target devices. Also, in such embodiments, the same output data may be broadcast to a number of target devices, so that, for example, multiple operations may be performed on the same data. In some embodiments, a single target peripheral device may receive data from multiple sources. In some embodiments, peripherals may send data to and receive data from a host CPU or host memory.

In some embodiments, a peripheral device configured for processing and transferring data may be used to perform a function that may have been offloaded from a host CPU. In some embodiments, a peripheral device may be added to a system via an expansion card. A new peripheral device may be intended to augment, supplement or replace an existing peripheral device and thus allow a system to perform a new function and/or perform an existing function in a more beneficial manner. In some embodiments, when a new peripheral device is added, a configuration step may be used to integrate the new device into a processing chain.

When appropriately configured, such peripheral devices may operate largely independently of a host CPU and may share data with each other to perform a chosen function (or functions). Embodiments may enable certain types of processing to be offloaded from a host CPU or a peripheral device to one or more other peripheral devices. Embodiments may offer more determinacy of execution time and superior real time performance.

Turning now to the figures,FIG. 1depicts an exemplary computer system82which may implement embodiments of the invention. Computer system82may be operable to execute the computer programs to implement computer-based systems and methods of data streaming described herein. The illustrated computer system82may include components such as CPU84and a memory medium86, such as random access memory (RAM), flash memory, hard-drives, and/or CD-ROMs. Memory medium86may include a storage medium having program instructions stored thereon, wherein the program instructions are executable to implement one or more embodiments of the present technique. The program instructions may be executable by CPU84, for instance. In the illustrated embodiment, computer system82includes a display device88, such as monitor, an alphanumeric input device90, such as keyboard, and a directional input device92, such as mouse. In one embodiment, computer system82may include modular and plug-in boards/cards (e.g., with either commercially available or proprietary hardware) that may be added via a number of expansion slots internal or external to the computer body (e.g., PCI or PCI Express slots). This system may be used for a wide variety of functions. For example, computer system82may be used for data acquisition (DAQ) when a DAQ digitizing board is plugged in and associated software is run.

FIG. 2Aillustrates an exemplary instrumentation control system100which may implement embodiments of the invention.FIG. 2Billustrates an exemplary industrial automation system160that may implement embodiments of the invention. The industrial automation system160may be similar to the instrumentation or test and measurement system100shown inFIG. 2A. Elements which are similar or identical to elements inFIG. 2Ahave the same reference numerals for convenience.

The illustrated system100may include host computer system82that may be configured to connect to one or more instruments/devices. In such a configuration, computer system82may operate with the one or more instruments to analyze, measure or control one or more devices or processes150. Computer system82may operate with the one or more devices in communication with process or device150to perform an automation function, such as MMI (Man Machine Interface), SCADA (Supervisory Control and Data Acquisition), portable or distributed data acquisition, process control, advanced analysis, or other control, among others. System100may be used in a data acquisition and control application, in a test and measurement application, an image processing or machine vision application, a process control application, a man-machine interface application, a simulation application, or a hardware-in-the-loop validation application, among others.

According to embodiments, one or more of the instruments and devices ofFIG. 2A(andFIG. 2B) may include a programmable hardware element, configured with an FPGA or a processor and memory, and may be further configured with one or more portions of user code, as will be described in more detail below. The one or more instruments may include a GPIB instrument112and associated GPIB interface card122, a data acquisition board114and associated signal conditioning circuitry124and boards126, a VXI instrument116, a PXI instrument118, a video device or camera132and associated image acquisition (or machine vision) card134, a motion control device136and associated motion control interface card138, and/or one or more computer based instrument cards142, among other types of devices. Further, the one or more devices may include a fieldbus device170and associated fieldbus interface card172, a Programmable Logic Controller (PLC)176, a serial instrument182and associated serial interface card184, or a distributed data acquisition system, such as the Fieldpoint system available from National Instruments Corporation, having headquarters in Austin, Tex., among other types of devices.

FIG. 3illustrates an exemplary computer system300that may implement embodiments of the invention. Computer system300may be similar to instrumentation system100or similar to industrial automation system160. It may also be noted that computer system300may be illustrative of a general-purpose desktop computer system, or a modular (e.g., racked) computer system held in a chassis, or the like. It may also be noted that many types of computer system configuration or architecture may be used as desired, and thatFIG. 3illustrates representative embodiments. Certain elements of the computer system that are not necessary to understand embodiments of the invention have been omitted for simplicity.

Computer system300may include at least one central processing unit (CPU) or “processor”84that may be coupled to bus304, for example, a front side bus. CPU84may be any of various types, including an x86 processor, a PowerPC processor, a CPU from the SPARC family of RISC processors, as well as others. A memory medium, typically comprising RAM and referred to as memory86may be coupled to bus304by means of a PC chipset (“chipset”)308. Chipset308may include one or more components that function together to provide some of the functions found in modern personal computers. In system300, chipset308may implement a memory controller, expansion bus controller and a host PCIe complex. As noted above, memory86may store a programming development environment, e.g., a graphical programming development environment, as well as one or more programs implementing and/or used in embodiments of the present invention. Memory86may also store operating system software, as well as other software for operation of computer system300. Computer system300may further include a video display subsystem310and hard drive312coupled to chipset308. In embodiments, components shown inFIG. 3as coupled to chipset308(e.g., display subsystems310and hard drive312) may be coupled to an expansion bus.

As depicted, in one embodiment, bus304may be coupled to an expansion bus314via chipset308. Expansion bus314may be a PCI (Peripheral Component Interconnect) expansion bus, although other bus types may be used. Expansion bus314may connect to card slots for the integration of various devices such as those described above. These devices may include GPIB card122and a GPIB Bus316, a MXI interface318coupled to a VXI chassis116, and/or a similar reconfigurable device320, for instance. In one embodiment, reconfigurable device320may include a processor and memory implemented on (e.g., configured on or included in), or coupled to, a programmable hardware element, such as an FPGA. Computer system300may be operable to deploy a program, e.g., a graphical program, to reconfigurable device320for execution of the program on reconfigurable device320, with respective portions of the program possibly implemented on the programmable hardware element, and stored in the memory for execution by the processor. Reconfigurable device320may be any of a variety of device types, such as those described above with reference toFIGS. 2A and 2B.

In some embodiments, a deployed program may take the form of program instructions, e.g., graphical instructions, or data structures that directly represent the program. Alternatively, the deployed program (or a portion of the program) may take the form of text code (e.g., “C” code) generated from a graphical program. As another example, the deployed program, or a portion of the program, may take the form of compiled code generated from either the graphical program or from text code that in turn was generated from the graphical program. Computer system82, e.g., via the programming development environment, may be operable to target (e.g., compile) respective portions of the user code for execution by the processor and for implementation as hardware on the programmable hardware element (e.g., the FPGA) as needed to meet performance criteria (e.g., resource use, timing, and I/O constraint criteria), and may be further operable to deploy the portions to their respective targets.

Computer system300may support PCI Express. In one embodiment, PC Chipset308may directly support a limited number of PCIe devices, but an expanded number of PCIe devices may be supported through the use of PCIe switches. In one embodiment, some (or all) of the attached PCIe devices may be integrated into computer system300using printed circuit board (PCB) cards or boards. PCIe devices may vary widely in terms of function. Device types may include input devices (e.g., video capture devices and digitizers), output devices (e.g., arbitrary waveform generators, or secondary storage interface devices), data processors (e.g., data processors used for analysis, test and control) and similar reconfigurable devices. Generally speaking, the types of devices integrated may reflect the intended use of system300.

Looking now at the exemplary embodiments of PCIe connectivity illustrated inFIG. 3, chipset308directly connects to device “A”330and device “B”332. Data may be transferred between devices330and332by flowing through direct connections made to chipset308. Chipset308also connects to PCIe switch350. Directly connected to switch350are device “C”334and PCIe switch352. Connected to PCIe switch352are device “D”336, device “E”338and device “F”340. PCIe switches350,352may make connections between connected devices so that, for example, device “E”338may send data to device “F” through switch352, without the data going to PCIe switch350. Note that, in computer system300, a wide variety of connections may be made through the two switches350,352and through chipset308. Note that with this topology, multiple data transfers may happen concurrently. For example, device “A”330may send data to device “D”336via chipset308and PCIe switches350,352. At the same time device “E”338may send data to device “F”340via PCIe switch352. An indirect transfer of data may refer to a transfer of data between devices that includes passing the data to or through a processor or memory of a host processing unit. With respect to devices of system300, transfers between devices (e.g., peripheral devices) that involve the transfer of data from one peripheral device to another peripheral device through a processor of host CPU (e.g., host CPU84) or through host memory (e.g., memory86) may be referred to as indirect transfers. Indirect transfers typically involve two stages, where stage one involves a transfer from a source device to a system host processor/memory and stage two involves a transfer from a system host processor/memory to a target device. An indirect transfer in system300, for example, may include a data transfer from device “A”330to device “C”334in two stages. In one embodiment, data is transferred from device “A”330to memory86(e.g., by device “A”330or by host CPU84) through chipset308and then data is transferred from memory86to device “C”334(e.g., by device “C”334or host CPU84) through chipset308and PCIe switch350. Such an indirect transfer may include a data transfer from a first peripheral device to a second peripheral device where the data is transferred through a processor and/or memory of a host CPU, such as from device330to any of the other devices332-340through a processor of host CPU84and/or memory86. A direct transfer of data may refer to a transfer of the data between devices without passing the data to or through a processor or memory of a host processing unit. Transfers from one peripheral device to another peripheral device through one or more buses and/or one or more links and/or one or more switches may be referred to as direct transfers, provided data is not routed through a system host processor/memory. In systems where a bus complex is integrated into the same device as a host CPU, transfers through the bus complex portion of the integrated device may still be considered to be direct, provided data is not routed through the processor or memory portion of the CPU device. In one embodiment a direct transfer may include data transfer from a first peripheral device to a second peripheral device via a PCIe Switch, such as from device “D”336to device “E”338via PCIe switch352. For example, a direct transfer in system300may involve device “A”330transferring data to device “C”334, through chipset308and switch350, without data routing through memory86or CPU84. This may be referred to as a direct transfer as data transferred via chipset308is not routed through a processor of CPU84and/or memory86. In some embodiments, host chipsets (e.g., chipset308) may include functionality similar to a PCIe switch (e.g., switch350, switch352). In such embodiments, data routed through chipset308may be routed through the PCIe switch portion and may not involve (or indeed pass through) the host CPU (e.g., CPU84) or host memory (e.g., system memory86) other host resources. In such embodiments, and in such cases, the transfer of data from one peripheral to another peripheral through chipset308may be considered to be a direct transfer as it does not invoke substantial functionality (e.g., processing functionality) of the host controller board or chipset, even though the transferred data may pass through a portion of the host controller board and host chipset. Embodiments similar to that described with respect to computer system300may have zero or more PCIe switches, and those PCIe switches may support different numbers of devices.

FIG. 4is a block diagram that depicts a computer system400in accordance with one or more embodiments of the present technique. Computer system400may be used for industrial purposes (e.g., test, instrumentation, process control), for instance. Depicted is an exemplary embodiment showing device (e.g., board) level PCIe connectivity and data flow between devices (e.g., between peripheral devices/boards). In the illustrated embodiment, system400includes a host controller board410having a CPU84, memory86, and chipset308. Other functions that may be found on the host controller board410are represented by the miscellaneous functions block406. Chipset308may provide PCIe links to a first peripheral device (“digitizer board”)411, a second peripheral device (“data processing board”)412and peripheral device interconnect (“PCIe switch”)350. In the illustrated embodiment, directly connected to PCIe switch350are a third peripheral device (“data processing board”)413and a peripheral device interconnect (“PCIe switch”)352. Connected to PCIe switch352are a fourth peripheral device (“data acquisition board”)414, a fifth peripheral device (“data processing board”)415and a sixth peripheral device (“arbitrary waveform generation board”)416. It will be appreciated that the type, number and arrangement of the peripheral devices is merely exemplary and are not intended to be limiting.

Embodiments of system400may be used in test and measurement systems, such as systems used to acquire, process or output data relating to a device or process being monitored or controlled, for instance. Accordingly, such a system may acquire data, and subsequently transfer the data for processing, storage, output, or the like. For example, in the case of system400being used to control a process (e.g., a closed loop system) data may be acquired, transferred to a processor where it may be processed, and subsequently transferred to an output such that the processed output may be employed to appropriately control the respective process. Often, streams of large amounts of data may be required to ensure data is being transferred at an appropriate rate. Moreover, where devices are used to implement several processes, certain channels for transferring data may include bottle-necks that slow the transfer of data or that otherwise inhibit the performance of the system. For this reason, in some embodiments, it may be beneficial to implement certain regulations relating to the transfers, and/or it may be beneficial to move data directly between the various peripheral devices (e.g., boards) that perform these different functions. For instance, the computer system400may support concurrent direct transfers of data between one or more of boards410-416via system connectivity.

In some embodiments, data may be transferred from a source (sending) device to a target (receiving) device of system400. The nomenclature of these devices may be used interchangeably based on the current role of the device (e.g., whether or not the device is sending or receiving data). In the illustrated embodiment, a first set of transfers is indicated by dashed lines420-422. In the illustrated embodiment, for instance, line420may represent data captured by digitizer board411that sent via chipset308to data processing board412for data processing (e.g., filtering to remove noise). Dashed line421may represent the transfer of data output from processing board412to board413for additional processing (e.g., filtering to remove artifacts). Dashed line422may represent the transfer of data from processing board413to host controller410(e.g., for storage of data in memory86). A second set of transfers, indicated by dashed lines430and431, may be performed independent of and/or concurrent with the first set of transfers420and421. In the illustrated embodiment, for instance, dashed line430may represent the transfer of data from DAQ board414to data processing board415(e.g., for analysis or processing), and dashed line431may represent the transfer of data from board415to arbitrary waveform generation board416(e.g., for outputting an analog waveform). Such transfers (430,431) may occur via PCIe switch352.

Note that parts of these transfers may happen concurrently. For example either430or431may occur concurrently with any of420,421,422and many other concurrent transfer combinations are possible. Note also that transfers420,421,430and431may be performed as direct transfers (e.g. transfers that do not route data through memory86. Transfers420and421, while they may route data through the chipset308, may route data through a host PCIe root complex contained in chipset308and may avoid routing data through memory86. Dashed lines440aand440bdepict an example of an indirect transfer (or pair of transfers) that may be performed as an alternative to transfer420. The first stage of this example indirect transfer (depicted by dashed line440a) is a transfer of data from digitizer board411through chipset308to memory86. This transfer stage may be performed by board411, by CPU84or by circuitry on host controller board410(e.g., chipset308). The second part of this indirect transfer (depicted by line440b) is a transfer of data (e.g., the data moved from board411) from memory86to processor board412. This transfer stage may be performed by processor board412, by CPU84or by circuitry on host controller board410(e.g., chipset308).

FIG. 5ais a block diagram representing a portion500A of a computer system, such as computer system300and/or computer system400. Portion500A contains three boards, namely, peripheral device board502, peripheral device board504and host controller board506. These (peripheral device) boards may be connected by interconnect switches507and508.FIG. 5ashows hardware features that may be found on a peripheral device board (e.g., board502) and data paths that may be utilized within a peripheral device board (e.g., board502).

In the depicted system, portion500A is exemplary in nature and that the depicted devices are not intended to infer that certain embodiments may rely upon (or are better suited to) certain illustrated components, classes of illustrated components, certain illustrated functions or combinations of illustrated functions. Further, peripheral device board502(and peripheral device board504) may be similar to one of the depicted devices “A-F” in system300and/or may also be similar to one of the depicted boards411-416in system400and described above with respect toFIG. 4. Additionally, host controller board506may be similar to host controller board410depicted in system400Further, interconnect switches507and508may be similar to PCIe switches350and352also depicted in system400and described above with respect toFIG. 4.

In some embodiments peripheral device board502may contain the following components, namely, input circuit510, output circuit512, input/output interface514, peripheral device516, data path switch518and system interface520. In some embodiments, input circuit510may includes an external input that allows data associated with an external system (e.g., another computer system, device-under-test) to be acquired. For example, in one embodiment, input circuit510may be a data acquisition analog input for receiving analog signals. In some embodiments, output circuit512may include an external output that allows data to be output from sub-system500(e.g., for transmission, display, storage or control). For example, in one embodiment, output circuit512may be a data acquisition analog output for outputting analog signals. The input/output interface514may be directly connected to input circuit510and to output circuit512and may serve to facilitate the transfer of data from input510and transfer of data to output circuit512. Interface514may be used to convert logic levels, convert signal types, change protocols or otherwise assist in interfacing board502to external devices or external systems.

In some embodiments peripheral device board502may be considered to be a peripheral board or a peripheral module or a peripheral device. However, board502may contain one or more sub-components that may be instrumental in implementing the function of the board. Peripheral device516represents such a sub-component (or sub-components). In one embodiment peripheral device516may include, for example, a processor, such as a digital signal processor or it may be a device customized to perform a dedicated function such as an application specific integrated circuit. In one embodiment peripheral device516may include, for example, a memory space, such as a FIFO buffer. During operation, peripheral device516may be used to process, store or perform other functions on the data. For example, processing device516may perform processing, such as filtering, of the data on board without having to transfer the data to another device, such as the host CPU or another peripheral device.

In some embodiments, peripheral device board502may also contain a system interface520that may be suitable for connecting board502to a computer system interconnect. The computer system interconnect may be a shared-bus interconnect (e.g., PCI) or a switched point-to-point interconnect (e.g., PCIe) or some other kind of bus (e.g., serial) or it may be an interconnect fabric. Data may be moved on and off peripheral device board502through the system interface520. Depending on the connectivity provided by sub-system500and connectivity of the host system that includes sub-system500, data may be moved from peripheral device board502to any other board in the system. System wide data movements will be explained in a description ofFIG. 5b.

In the illustrated embodiment, switch518may be used to route data on peripheral device board502. Switch518may route data (e.g., data from input circuitry510or data from peripheral device516) through system interface520to one or more destinations in the system. For example, data may be routed to host controller506via switches507and508. Alternatively, data may be routed to peripheral device board504(which could include a processor, for example) via switch507. In such an embodiment, data may be passed directly, thus bypassing switch508and host controller506. Switch518may route data (e.g., data from system interface520) to peripheral device516or to output circuitry512. In one embodiment, this data may be routed from host controller506via switches508and507. Alternatively, the data may be routed from peripheral device board504via switch507.

In some embodiments, data may enter peripheral device board502from various locations, and data may leave peripheral device board502to various locations, as previously described. In one embodiment, switch518on peripheral device board502may facilitate data flow from any appropriate source on peripheral device board502(e.g., input circuitry510and/or peripheral device516) to any appropriate destination, whether the destination is on the peripheral device board (e.g., peripheral device516, output circuitry512) or the destination is elsewhere in the system. System destinations may include a peripheral device (e.g., such as those having a processor) on another peripheral board (e.g., co-processor board504) or a device on a host controller board (e.g., host controller506).

In the illustrated embodiment, data may flow from input circuit510to host controller506as follows. Data may flow from input circuitry510to input/output interface514via connection540A (connection540B may be used for output), to internal switch518via connection540C, through internal switch518and to system interface520via connection540E. From system interface520, data may flow to switch507via connection540F, through switch507, to switch508via connection542E, through switch508and to host controller506via connection542H. In the illustrated embodiment, data may also flow from peripheral device board504to peripheral device board502as follows. Data may flow from peripheral device board504to switch507via connection542G, through switch507, to system interface via connection540F, to internal switch518, through internal switch518and to peripheral device516via connection540D. In some embodiments, multiple data flows involving the same boards, same devices and same switches may happen concurrently. For example, in the illustrated embodiment, data may flow from input circuit510to host controller506while flows from co-processor board504to peripheral device516.

In some embodiments, data received by a peripheral device (e.g., peripheral device516of board502) may be processed by the peripheral device to produce processed data that may be sent by the peripheral device. Peripheral devices (e.g., co-processors) may be used to perform processing on behalf of other devices (e.g., other peripheral devices or host CPUs). In some embodiments, peripheral devices may be “chained” together to perform a sequence of processing functions. In some embodiments a peripheral device may perform a function (e.g., sample an analog signal, communicate with test equipment, read data from storage) or a portion of a function (e.g., filter a portion of an image, perform an initial analysis) or a combination of functions (e.g., decompress an image and format it for display). Also, the output of a processing step/sequence may be sent to one or more peripheral devices in a processing chain. In some embodiments, data transfers between devices may be performed directly (e.g., without routing data through the host CPU or host memory).

FIG. 5bis a block diagram representing a portion500B of a computer system that may be similar to computer system300and computer system400described previously. Portion500B comprises four boards, namely, peripheral device board502A, peripheral device board502B, peripheral device board502C and host controller board506A. These boards may be connected by interconnect switches507A,507B and507C.

In some embodiment, boards502A,502B and502C may be similar to peripheral device board502depicted in and described with respect toFIG. 5a. As previously mentioned, peripheral device boards may fulfill a variety of functions and consequently, the devices on a certain board may reflect the function (or functions) of the board. In the illustrated embodiment, peripheral device board502A represents a peripheral input board with depicted input circuitry510A, input interface514A and input peripheral device516A (and no depicted output devices). Peripheral device board502B represents a co-processor peripheral board with depicted processor peripheral device516B. Peripheral device board502C represents a peripheral output board with depicted output circuitry510C, output interface514C and output peripheral device516C. Those skilled in the art will appreciate that there is a wide variety of peripheral boards having various configurations. For example, a peripheral board may include one or more input circuits, one or more output circuits, one or more peripheral devices, or the like.

In some embodiments, such as those depicted by500B, board502A connects to switch507A which, in turn, connects to board502B and switch507B. Switches507A,507B and507C may be similar to switch507depicted inFIG. 5A. Switch507B connects to switch507C and also to host controller506A that may be similar to host controller506depicted in and described with respect toFIG. 5A. Lastly, switch507C connects to peripheral device board502C.

In some embodiments, such as those depicted by500B, peripheral device boards502A,502B and502C may form one or more processing chains in which data may be transferred between peripheral components on the boards via internal bus switches (e.g.,518A,518B and518C) and external switches (e.g.,507A,507B and507C). In certain embodiments, the transfer of data between peripheral boards/devices may be performed directly, that is, without routing data through the host CPU or host memory.

In some embodiments, such as those depicted by500B, a first processing sequence may operate as follows. Data may be input (e.g., from sampling equipment) via input circuitry510A and may flow to input peripheral device516A through input interface514A and internal bus switch518A. The input peripheral device516A may process none, some, or all of this input data (e.g., perform a filtering operation) and may send output data to output peripheral device516C via elements518A,520A,507A,507B,507C,520C and518C. The output peripheral device516C may process none, some, or all of the data sent from516A to produce new output data (e.g., equipment control messages) and send this new data via internal bus switch518C, output interface514C to output circuitry510C (e.g., to output a signal for controlling equipment). In the first processing sequence, peripheral board502A (e.g., input board) exchanges data directly with peripheral board502C (e.g., output board) using switches507A,507B and507C. The first processing sequence may, for example, be used to analyze and control equipment and may perform this function without routing data through host controller board506A. In such an embodiment, a first peripheral device may directly transfer data to a second peripheral device, and the second peripheral device may process at least a portion of the data using an onboard processor.

In some embodiments, a second processing sequence may be employed when, for example, additional processing of data may be beneficial. This second processing sequence may be performed in a similar manner to the first sequence, subject to appropriate system configuration. In some embodiment, the system may be dynamically configurable. Dynamically configurable may refer to the system being capable of having its configuration modified, e.g., by a user at of before run time. In some embodiments, dynamically configuring the system may include specifying/modifying the paths/routes for data transfers to occur and/or specifying/modifying how and where processing may occur. The user of an embodiment may determine the processing functions to be performed by the embodiment and the user may also determine the devices/boards to be used, perhaps through an enabling application. After determining the function to be performed and devices to perform the function, the system may (perhaps with user input) configure peripheral devices, peripheral boards and link switches so that the employed devices may transfer data to from one device to another to offload processing between the devices. This configuration may involve sending configuration information to entities such as devices or boards or switches. On receiving such configuration information, entities may be configured or configure themselves to perform the intended function and share data in the intended fashion.

In the second processing sequence (or processing chain), data may be routed to/from peripheral device board502B for processing by processor peripheral device516B. This second processing sequence may operate as follows. Data may be input (e.g., from sampling equipment) via input circuitry510A and may flow to input peripheral device516A through input interface514A and internal bus switch518A. The input peripheral device516A may process this input data (e.g., perform a filtering operation) and may send output data to processor peripheral device516B via elements518A,520A,507A,507B,520B and518B. The processor peripheral device516B may process this received data (e.g., perform pattern recognition) and generate output data (e.g., simplified 2D representation). Processor peripheral device516B may send a portion of the data it generated to output peripheral device516C via elements518B,520B,507A,507B,507C,520C and518C. In one embodiment, at least a portion of the data sent from516B may be processed by output peripheral device516C to produce new output data (e.g., equipment control messages) and send this new data via internal bus switch518C, output interface514C to output circuitry510C (e.g., for outputting a signal to control equipment). The second processing sequence may, for example, be used to analyze and control equipment and may perform this function without routing data through host controller board506A. In one embodiment, a portion of the data may be sent from device502B back to board502A via elements518B,520B, and507A (e.g., for additional processing or output where board502A includes an output, for instance).

Certain embodiments may incorporate sub-systems similar to the depicted subsystem500B. These embodiments may include multiple peripheral boards (e.g., co-processor boards) that are similar to peripheral device board502B. In such embodiments, multiple peripheral boards may operate together to perform a function. Data may be sent from an input peripheral device board (e.g., similar to502A) to multiple peripheral boards. In some embodiments, a certain portion of data may be sent to multiple boards. For example, in an embodiment with four co-processor peripheral device boards, an entire image may be sent, perhaps in broadcast fashion (e.g., in parallel), to each of the four co-processor boards, so that, for example, each device could perform a separate image processing function on the image. Transferring data in parallel may refer to transferring data from one device to multiple devices non-sequentially, e.g., from device A to device B and from device A to device C. Parallel transfers may or may not occur simultaneously. Transferring data in series may include transferring from one device to multiple devices sequentially, e.g., from device A to device B and from device B to device C. In some embodiment, processing may occur in parallel, e.g., in two device may process data independent of one another. For example device A and device B may both process data independent of one another, and the processing may or may not occur simultaneously. In one embodiment, processing may occur in series, e.g., sequentially in two or more devices. For example, device A may process data and passing the processed data to device B for subsequent processing. Alternatively, in an embodiment with four co-processor peripheral device boards, data for a quarter of an image may be sent to each of the four co-processor boards, so that, for example, each device could perform image processing on a quarter of the image.

FIG. 6depicts a flow chart of an exemplary method600offloading data to another device for processing, in accordance with one or more embodiments of the present techniques. In some embodiments method600starts with a configure device step as depicted by block602. This step may involve configuring one or more devices in a computer system, in accordance with any of the techniques described herein. For example, a configuration may dictate that data be distributed from one device to another for processing, in accordance with any of the techniques described herein as well as any potential path with the described system. A device may be configured automatically by a host system and it may be configured with user input that, in turn, may be obtained from a human interface device. Some embodiments may display information (e.g., diagrammatic information) relating to a task, process or function and some embodiments may also display information about one or more system devices that may be involved in a task, process or function. A system user may read such displayed information and may select devices and configure those devices (e.g., configure how data is shared between the devices) for a chosen task or tasks. Device configuration may involve, for example, arranging for a device to perform a particular task (e.g., filtering an image) or a portion of a particular task (e.g., filtering a portion of an image) or a group of tasks (e.g., filtering an image and compressing an image). In one embodiment, the transfer paths of data may be configured. For example, a device may be configured to receive data from a particular device (e.g., an input device such as a digitizer), a portion of data (e.g., a quarter of the data) from a particular device or portions of data from various devices. A device may also be configured to send a portion (e.g., 100%, 25% or 10%) of data to a particular device (e.g., a processing peripheral device). Additionally, a device may be configured to broadcast data to multiple devices. For example, in one embodiment, a device may be configured to send an image to two co-processing devices in parallel so that two different filters may be performed on the same image. A device may be configured to send data to various types of devices for a variety reasons. For example, a device may be configured to send data to an output device for display, to an output device for control, to a host controller for storage, to a communications device, to a reconfigurable device for processing or to an inter-mediate device for forwarding to a destination device and so on.

In some embodiments, data may be sent from one device to another device via point-to-point links that may be connected by one or more switches (e.g., PCIe switches). In some embodiments a switch (or switch functionality) may reside on a system chassis or on a host controller board or on a peripheral device expansion board. Switch functionality may be integrated onto devices such as PC chipset devices, controllers, interface circuits, bus converters or peripheral devices. Since switches may affect the routing of data, configuring the system for sharing data may also, in some embodiments, involve configuring one or more switches.

In some embodiments method600may include a receive data step as depicted by block604. This step may involve a device receiving data from one or more sources. Data may be received from various types of devices. Depending on the embodiment, data may be received from an external input, such as that of an input device (e.g., a digitizer, video source), data may be received from an internal input, such as directly from another peripheral device (e.g., a peripheral co-processor device) or indirectly via/from a host device (e.g., a system host controller). In some embodiments, a device may receive a portion of data from a number of devices. For example, in one embodiment, an image processing device may receive the upper half of an image from a first device and the bottom half of an image from a second device.

In some embodiments method600may also include a send data step as depicted by block606. This step may involve a source device sending data to one or more target devices. Data may be sent to various types of devices. Depending on the embodiment, data may be sent to an output device (e.g., a graphics display controller) or data may be sent to a processor device (e.g., a peripheral co-processor device) or data may be sent to a host device (e.g., a system host controller), perhaps for storage purposes. In some embodiments, a device may send data to a number of devices. For example, in one embodiment, an input device may send the upper half of a captured image to a first device (e.g., for filtering) and the bottom half of the captured image to a second device (e.g., for filtering). In certain embodiments, data may be sent in a broadcast fashion (e.g., sent to multiple targets, perhaps simultaneously). For example, in an embodiment where three different image processing filters are to be performed using three separate processing devices, an image capture device may broadcast data for one image to the three devices.

Depending on the embodiment, the data that is sent may be partially processed or unprocessed or fully processed. The sent data may be related to data that was previously received or it may be a portion of received data or it may be unrelated to received data. In some embodiments, data may be sent from a first device to second device and then data may be sent from the second device to a third device and so on and so forth, so that a sending and receiving chain of devices is formed.

In some embodiments method600may also include a process data step as depicted by block608. The process data step608may depend on the embodiment and depend on the processing device (or devices). For example, step608may involve a device fully processing a portion (e.g., 100%, 50%, 10% etc.) of the data to be processed and it may involve a device partially (e.g., 100%, 25%, 5% etc.) processing a portion of the data to be processed. Processing may be performed, for example, by input devices, co-processor devices, reconfigurable devices, dedicated devices, custom devices or general purpose processing devices and so on. In some embodiments, multiple devices may be used to perform one processing step. In certain embodiment, this may be achieved, for example, by dividing data among multiple processing devices or it may be achieved by dividing the processing step among multiple processing devices. In some embodiments, some devices may be involved in the processing of data but may not actually process data. For example, some devices (e.g., input devices) may acquire data and send the acquired data to another device which may process the sent data.

In some embodiments method600may also include an output data step as depicted by block610. Data may be output for a variety of purposes. For example, data may be output for display purposes or for storage or for external control or for transmission or for other purposes. Various types of data may be output, for example, output data may include partially processed data and/or fully processed data. Depending on the embodiment, data may be output by multiple devices or by a single device.

The methods illustrated by flow chart600may include steps602,604,606,608and610in the depicted order. However, those skilled in the art will appreciate that in some embodiments some of these steps may occur in a different order. For example, in some embodiments, receiving devices may process data before sending data. Those skilled in the art will also appreciate that in some embodiments (e.g., data acquisition systems) a portion of the depicted steps (602-610) may essentially occur concurrently and a portion of the depicted steps may also occur repeatedly

With respect toFIGS. 1 through 6, various embodiments relating to the transfer of data have been depicted and described. For example, embodiments that support the direct transfer of data between peripheral devices without routing data through a host controller have been described. Those skilled in the art will therefore appreciate that embodiments of the invention may benefit from an efficient mechanism for transferring data between peripheral devices.FIGS. 7 through 10, together with accompanying text, describe embodiments of a mechanism for such peripheral device to peripheral device data transfers.

FIG. 7is a block diagram representing an exemplary portion700of system400illustrated inFIG. 4. In the illustrated embodiment, exemplary portion700includes three devices: data acquisition board414, data processing board415and data output board416. Note that other embodiments may have different combinations of boards and different types of devices. For instance, embodiments may include combinations of peripheral devices or combinations of one or more host devices and peripheral devices. In certain embodiments, one or more portions of the devices) may be integrated onto a single device, such as multiple PCIe devices and/or PCIe switches integrated into single modular peripheral device. Embodiments of the invention are not limited in terms of the physical design of the computer system or in terms of the functions of the PCIe devices.

In one embodiment, DAQ board414may include an input device that receives data, and thus may be a source of data to be processed and output by sub system700. For example, DAQ board414may include a digitizer that receives an analog electrical signal, and digitizes the signal into computer readable data (“data”). In one embodiment, the data may be stored or processed on DAQ board414. In one embodiment, the data may be transferred to another portion of the computer system for storage at another location and/or processing at another location within the system. For example, the data may be transferred to the system host (such as host controller410, which is not depicted but may be connected via PCIe link762) and/or another peripheral device, such as data processing board415. The transfer of data may include “indirect” or “direct” transfers of the data between devices. As discussed briefly above, in order to allow the system to operate reliably it may be beneficial to implement devices and techniques to regulate the transfer of data between devices, such as the streaming of data between devices.

In the illustrated embodiment, DAQ board414may include a buffer (“FIFO”)702for managing the flow of data being captured and being transferred. Data may be transferred using direct memory access controller (DMAC)708which may be programmed to automatically transfer data. In one embodiment, DMAC708may be programmed by transfer control logic703. The transfer control logic703may determine the characteristics data transfers including, whether or not a transfer should be initiated. Transfer control logic may be coupled to FIFO702.

As depicted inFIG. 7, DAQ board414may also include two registers, a stream additive transfer count register (SATCR)704and a total transfer count register (TTCR)706. SATCR704may include a mechanism used to control the initiation of data transfers. For example, SATCR704may be associated with and/or monitor a SATCR value indicative of whether or not a source (“sending”) device should initiate a transfer of data to a target (“receiving”) device. As discussed in more detail below, the SATCR value associated with SATCR704may be indicative of the number of bytes that a receiving device is able to receive. For example, the SATCR value may be written by software and/or hardware and may be adjusted (e.g., decreased/increased) by hardware as data transfers are initiated by board414. In one embodiment, the amount that the SATCR value is decreased/increased may correspond to the byte count of the data transfer being initiated. In one embodiment, transfer control logic703may be configured to initiate transfers only as long as the SATCR value is above a certain threshold, thereby helping to ensure that the receiving device is capable of receiving the amount of data being sent. In other words, control logic703may be configured to not initiate data transfers that may cause the value of the SATCR704to drop below a threshold (e.g., zero) to help ensure that there is enough space for the data at the receiving device, thereby helping to prevent data over-run at the receiving device. Note that, depending on the embodiment, the value of transfer credits may be represented in different ways. For example, in one embodiment, an increase in the value of transfer credits may equate to an increase in a corresponding number. In another embodiment, an increase in the value of transfer credits may equate to a decrease in a corresponding number. Transfer credit threshold values may be set according to the embodiment and according to the representation of transfer credits used in that embodiment. So, for example, in some embodiments the value of transfer credits dropping below a threshold may actually correspond to an associated number rising above a threshold.

Due to the latency involved, the performance of sub system700may be impacted if software were to programmatically pause data transfers in order to write/update a new value to SATCR704. Such a pause may be attributed to software writing a control bit to pause hardware transfers, reading the current SATCR value of SATCR704, and then writing an updated value before un-pausing the hardware. The pausing transfers may be expected by those skilled in the art since the SATCR may be adjusted both by circuitry (e.g., as data is transferred) and also by software writes (e.g., by a target device granting more transfer credits). Since it is possible that two adjustments may be made coincidentally, some kind of adjustment error or problem may be anticipated. However, in some embodiments the SATCR may have associated circuitry that prevents problems arising from a coincidence of software and hardware adjustments. In one embodiment, values written to SATCR704may be automatically added to the current value via hardware. Such a method may allow writes (e.g., originating in hardware or software) to continually increase the amount of data that hardware is enabled to transfer by simply writing directly to SATCR704for each update. In such an embodiment, SATCR704may be updated via communication between two hardware devices. Thus, updates may occur before the register reaches zero and hardware may be stalled only when desired, such as when synchronizing hardware with software. In such an embodiment, a sending device (e.g., board414) may monitor the SATCR value of the SATCR704which has been updated based on signals/values received directly from a receiving device (e.g., board415) depicted in the dotted line750, such that the two peripheral devices may regulate the transfer of data between them without transferring such flow control information using a host device or through host controller memory.

When using a SATCR704as described above, register accesses from software may be reduced to a single write per target device update. Further, transfers (e.g., DMAC-based transfers) may be decoupled from software's flow control mechanism such that unnecessary stalls are reduced.

In one embodiment, TTCR706may be used to track the amount of data sent/transferred by a device. For example, TTCR706may include a readable status register that may provide a total byte count of the data that has been transferred since the beginning of one or more DMA operations. In some embodiments, reads of this register are coherent such that the value returned may be immediately acted upon without any further synchronization steps. For example, a TTCR of a sending device, such as board414, may be read to determine how much data has been sent to a receiving device, such as board415. The coherency of a TTCR register may enable the data associated with TTCR to be read by the receiving device with minimal or no further synchronization steps. The coherency also allows the sending device to immediately free or reuse locations associated with the data that the TTCR indicates has been sent.

In some embodiments, system400and/or sub-system700may implement a technique known as “DMA Progress Status Pushing” (DPSP). Such a technique may include automatically writing (e.g., via hardware) the TTCR value (e.g., TTCR706) to a specific location where it may be read. This location may be in a memory (e.g., memory86) or it may be in a memory or a register on the target device (e.g., board415), for example. The hardware may write every time a predetermined (e.g., fixed and/or programmable) amount of DMA data bytes have been transferred. This may avoid some costly hardware register reads of TTCR and may allow driver software to monitor DMA progress by reading the information from a more convenient location. Like reads of TTCR referred to in the previous section, in one embodiment, a status push of TTCR is guaranteed to be coherent with the data being sent to the same destination. The reader of the status pushed TTCR may act immediately on the TTCR value. The hardware may also be capable of “pushing” (e.g. writing) the TTCR value at the end of a transfer operation since the end may not coincide with a multiple of the programmed DPSP byte count. A bit in the pushed value may be used to distinguish between pushes that were triggered by the programmed byte count being transferred and the push at the end of an operation.

Both reads of TTCR and DMA Progress Status Pushing may be employed in situations where a controller desires to query DMA progress. In one embodiment, interrupts may notify a controller when DMA progress is made. For example, in one embodiment, a technique referred to as “Every N Notification” (ENN) may be used to interrupt a controller each time a fixed and/or programmable number of bytes have been transferred. In one embodiment, this feature may be combined with DMA Progress Status Pushing so that, for each interrupt, the controller may obtain updated status about DMA progress without performing a read of a TTCR on the sending device. As with DMA Progress Status Pushing, the control hardware associated with ENN may also generate an interrupt at the end of a transfer operation since the end may not coincide with a multiple of the programmed ENN byte count.

Some embodiments may involve interrupting a controller (e.g., a host CPU or a peripheral controller) when a specific number of bytes have been transferred. Instead of interrupting on a regular basis a device controller may wish to specify a particular byte count and receive an interrupt at that point. The number of bytes that the device controller specifies may vary during operation. A Total Transfer Count Compare Register (TTCCR) may allow a controller to specify a total byte count at which an interrupt is desired. In one embodiment, the TTCCR may be hardware based. In such an embodiment, hardware may generate an interrupt when an associated TTCCR reaches a value associated with the specified byte count. After receiving the interrupt the controller may be free to specify another value for subsequent notification.

Some embodiments may support a feature with similar functionality to ENN. Certain embodiments may include additional circuitry on sending and receiving devices. In one embodiment, a technique referred to as “Every N Doorbell” (END) may include a hardware initiated write of a programmable data value to a specific address every time a predetermined (e.g., fixed or programmable) number of bytes have been transferred. The specific address may correspond to a register location on a receiving device. In one embodiment, hardware on the receiving device may, on detecting a write to the register at the specific location, cause an interrupt to be sent to an associated controller. In certain embodiments (e.g., embodiments with a system interconnect based on PCIe) doorbell functionality may be advantageous if the bus protocol associated with system interconnect provides limited dedicated or explicit means for one peripheral device to interrupt another peripheral device. In such embodiments, doorbell functionality may be advantageous even when the system interconnect provides an interrupt mechanism directed at a host CPU.

A register that generates an interrupt when the register is written may be referred to as a “doorbell”. Some embodiments may include a doorbell register. In one embodiment, a write to a doorbell register may be coherent with transmitted data at a target device. For example, in one embodiment, a doorbell may generate interrupts to a co-processor target device that may be coherent with received data in the co-processor's memory. In one embodiment, DMA Progress Status Pushing may be combined with the END technique so that the coprocessor may quickly retrieve a progress status from its local memory with each doorbell interrupt. In one embodiment, a doorbell write may also be initiated at the end of a data transfer operation where the quantity of data transferred may not be a multiple of the predetermined N byte count. Similarly, in one embodiment DMA Progress Status Pushing may occur at the end of a data transfer operation and the pushed value may include a bit that may be used to distinguish between a push that was triggered by a multiple of N and a push that was triggered by the end of the data transfer operation.

In one embodiment, a first value in a TTCCR may be over written with a smaller second value before an interrupt corresponding to the first programmed value is generated. Such a technique may allow a peripheral controller, for example, to be notified at an earlier point in the data stream than may have been previously specified. Some embodiments may include preventing the overwriting of a first value in a TTCCR with a larger second value before an interrupt corresponding to the first programmed value is generated. This may help to avoid a race condition. In such embodiments software may be restricted to waiting until a previously programmed interrupt is received before trying to program a new value.

Looking now at the processing of data and the movement of data between devices, in one embodiment, multiple devices, such as the three boards (414,415and416) of sub-system700, may form a processing chain. For example, the devices may acquire data, processes data, and generate an output. Using techniques, such as those described herein, large quantities of data (e.g., many giga bytes or tera bytes) may be moved between the boards as the processing steps in the chain are performed. In addition, sub-system700may perform a large number of simultaneous data movements during operation within system400.

As discussed above, in one embodiment, DAQ board414may include an input device that receives data, and thus may be a source of data to be processed and output by sub system700. For example, DAQ board414may include a digitizer that receives an analog electrical signal, and digitizes the signal into computer readable data (“data”). In one embodiment, the data may be stored or processed on DAQ board414. For example, the data may be stored in FIFO702of DAQ board414in preparation for sending to data processing board415. In one embodiment, the data transferred to DAQ board415is transferred into an input FIFO710. In one embodiment, when there is adequate space in input FIFO710of board415, board415may write (as indicated by750) to SATCR704, in order to grant board414the ability to transfer data. As referred to herein, “transfer credits” may be used to describe the amount of data the receiving device has allotted or otherwise given permission for the sending device to transfer to the receiving device. In one embodiment, credits may include a number of bytes a sending device may transfer to the receiving device based on free space available in a buffer of the receiving device. For example, where the receiving device has 128 bytes of free space and a transfer credit is indicative of 1 byte, 128 transfer credits may be provided to the sending device to indicate that it may send 128 bytes to the receiving device. During use, multiple transfer credits and/or sets of transfer credits may be issued to the sending device such that they are accumulated. The total number of transfer credits currently available to the sending device may be referred to as the “accumulated transfer credit value.” The total accumulated transfer credit value may be increased as additional transfer credits are issued from the receiving device, and reduced as data is sent, using at least a portion of the issued transfer credits.

In one embodiment, transfer control logic703may determine that sufficient transfer credits have been granted for a data transfer to take place. In one embodiment, if it is determined that sufficient transfer credits have been granted, transfer control logic703may enable DMAC708to transfer data to data processing board415, for example. DMAC708may then transfer data to input FIFO710. In one embodiment, the transfer of data may occur directly via a PCIe point-to-point link connected at PCIe switch352. In one embodiment, DAQ board414may write (as indicated by dashed line752) data transfer information (e.g., the value of TTCR706) to location Input Bytes Received (IBR)715to communicate transfer progress to board415. Data processing board415may use the value of IBR715to determine how much data it has received. Data processing board415may determine the quantity of received data and then may read the exact or an associated amount of data out of input FIFO710for processing, freeing up space in FIFO710as the data is read. As board415performs these actions it may grant more data transfer credits (by writing to SATCR704as indicated by dashed line750) for the newly freed space in FIFO710. Thus, as space is freed in the FIFO of the receiving device (e.g., board415), the receiving device may dynamically issue transfer credits to the sending device414. Transfer control logic703may monitor SATCR704for additional credits and if it determines that sufficient additional credits have been granted, it may send additional data to DMAC708to be transferred to board415. Such a technique may enable a stream of data to be transferred between the devices.

In one embodiment, data processed by board415may remain on board415and/or it may be transferred to another device or location for processing, storage, output, or the like. In one embodiment, for instance, data processed by board415is placed in output FIFO712in preparation for transfer to data output board416for output (e.g., via an external connection to a DUT or process150as described with respect toFIGS. 2A and 2B). Similar to the above described embodiments, data output board416may determine if there is sufficient space in FIFO720and may grant transfer credits accordingly to board415by writing (as indicated by dashed line756) to SATCR714. Transfer control logic713on data processing board415may determine that the accumulated transfer credit value is sufficient for a data transfer and enable the DMAC718to transfer data to FIFO720. The DMAC718may transfer data by sending data directly to FIFO720of output board416via PCIe switch352. In one embodiment, data processing board415may write (as indicated by dashed line754) data transfer information (e.g., the value of TTCR716) to location Input Bytes Received (IBR)725to communicate transfer progress to board416. Data output board416may read/output data from FIFO720(e.g., to an output) and, as data is read, more space is freed in FIFO720, thereby enabling data output board416to grant additional transfer credits to data processing board415(as indicated by756). In response to receiving more transfer credits from data output board416, the accumulated transfer credit value may increase. This may, in one embodiment, enable transfer control logic713to transfer more data from output FIFO712to DMAC718to be transferred to FIFO720.

Turing now toFIG. 8, depicted is a flow chart of an exemplary method800for transferring data between a source device and target device, in accordance with one or more embodiments of the present technique. As depicted at block802, method800may include determining available space. For example, in one embodiment, method800may include a target peripheral (e.g. one that may be configured to receive data), determining how much storage space is available (e.g., space available in a FIFO or in a RAM) for receiving data. In one embodiment, the determining of available space may be performed by a device acting on behalf of a target device, such as a controller associated with the target peripheral or a host CPU.

In the illustrated embodiment, method800also includes sending transfer credits, as depicted at block804. In one embodiment, the target device (or a device acting on behalf of the target device) may send transfer credits to a source device that may be configured to send data. This may occur after determining available space at block802, as depicted in the illustrated embodiment. The amount/value of transfer credit sent by the target device may depend on various factors. For example, in some embodiments, the number of transfer credits issued may be based on the amount of available space determined at block802, the amount/value of transfer credits previously sent, and/or the quantities data previously received and/or processed at the receiving device. In one embodiment, the number of transfer credits issued may be the same or at least based on the amount of free space available in a buffer of the receiving device. For example, in one embodiment, if a receive FIFO has 256 bytes of storage space for receiving data and 128 bytes have already been received, the receiving device may grant an additional transfer credit of value 128 bytes.

In the illustrated embodiment, method800may also include receiving transfer of credits, as depicted at block806. In one embodiment, a source device may receive transfer credits sent at block804. For example, a target device may grant a transfer credit of value 4096 bytes, by writing a value corresponding to 4096 bytes to a specific location on a source device. In the illustrated embodiment, method800also includes enabling a quantity of data to be transferred as depicted at block808. In one embodiment, the quantity of data enabled to be transferred may correspond to an accumulated transfer credit value. In one embodiment, a source peripheral may enable a quantity of data (e.g., based on the accumulated transfer credit value) to be transferred to a target device. In one embodiment the enabling process may involve configuring a DMAC to perform data transfers. In one embodiment, enabling may involve allowing transfer requests to be sent to a DMAC.

In the illustrated embodiment, method800further includes sending data and reducing the current value of transfer credits as depicted at block810. Note that, in some embodiments, data may be sent in accordance with the operation of a bus governor. In such embodiments, a bus governor may be used to limit the utilization of a bus that a particular device or particular module or particular board may obtain over a specified period of time. In one embodiment, a bus governor may be controllable by software and may allow software to specify a level of bus utilization that a device may obtain and may also allow software to specify the interval over which bus utilization is measured and controlled. For example, a bus governor may be used to specify a rate, duration, period, and/or frequency at which a device may transfer data across a bus. In one embodiment, a bus governor may operate on a shared backplane bus (e.g., PCI) of a computer system. In another embodiment, one or more governors may operate on one or more point-to-point links, such as is found in PCIe-based systems. Such a technique may help to ensure that certain devices do not dominate a shared bus (or shared link), thereby preventing other devices from sufficient access to the shared bus.

In one embodiment, the current value of transfer credits may be reduced according to the quantity of data sent. For example, a source device may send data to a target device and may decrement the allocated transfer credit value according to the quantity of data sent. In one embodiment, the transfer credits and/or accumulated transfer credit value may be stored in a counter, and the counter may be progressively decremented as each data item is transferred to the receiving device.

In one embodiment, the transfer of data may be repeated substantially continuously, thereby providing streaming data transfer. However, at some point the source device may, as depicted at block812, stop sending data. Stopping may include a pause in sending during the streaming of data, and/or may include an end to the streaming of data. In one embodiment, the source device may stop sending data if the accumulated transfer credit value falls below a threshold (e.g., the accumulated value of transfer credits falls to zero). In some embodiments, stopping the sending of data may involve inhibiting programming of transfer requests into a DMAC.

FIG. 9is a flowchart that depicts an exemplary method900for transferring data in accordance with one or more embodiments of the present technique. Such a technique may be performed by a source device. Moreover, it will be appreciated that method900may be implemented in conjunction with any of the techniques described herein, such as those described with respect toFIG. 8. In the illustrated embodiment, method900may include determining whether or not there is new data to send, as depicted at block902. For example, the source device may first determine if there is data to be sent. This may involve determining if there is initial data to be sent and/or may involve determining if there is new data to be sent. For example, it may be determined whether or not data is stored in a buffer/FIFO of the sending device. If there is no new (or initial) data to be sent the sending device may wait until such data is available, repeating step902as illustrated.

In the illustrated embodiment, if it is determined that there is new data to be sent, before starting to send data, the sending device may first determine if the accumulated transfer credit value meets or exceeds a threshold, as depicted at block904. For example, in one embodiment the sending device may determine if the current value of transfer credits is greater than zero. In some embodiments a default value of transfer credits may be automatically granted upon initialization. If the current value of transfer credits does not meet a threshold, the sending device may wait for a target device to grant a sufficient number of transfer credits, as depicted. If there are enough transfer credits, the source device may then start to transfer data to the target device. In one embodiment, the sending of data may be performed by one or more of the source device, a DMAC coupled to the source device, a DMAC module on the source device or a bus master associated with the source device. For ease of explanation, sending may be described hereafter as sending by the source device. InFIG. 9, the sending is depicted as being contained within an iterative loop comprising elements904,906,908and910.

In one embodiment, if it is determined (either prior to sending or during sending), that the accumulated transfer credit value meets/exceeds a threshold (block904) then the source device may transfer one or more data items to the target device, as depicted at block906. The size of each data item may depend on system characteristics such as the nature of bus connecting the source and target devices.

In some embodiments, determining if the accumulated transfer credit value meets/exceeds a threshold may include determining if there is sufficient data for a specific size/type of data transfer. The ability to stream large quantities of data across a bus may depend upon the utilization of the bus. The utilization of the bus may, in turn, depend on the type of bus transfers performed and may also depend on the quantity of data associated with each data transfer. In buses such as PCIe, each packet may have a fixed amount of overhead, so sending small packets may not make optimum use of the available bus bandwidth. Furthermore, sending larger packets to aligned buffers in memory may make better use of the available target memory bandwidth.

Sending and receiving devices (e.g., peripherals) may make better use of transfer bandwidth of a connecting bus (or some other type of link) by using a transfer quantity value known as Preferred Packet Size (PPS). In some embodiments, the sending of data may be delayed until a quantity of data corresponding to the PPS is available for sending. Then, when at least a PPS quantity of data is available, data of PPS size may be sent. Similarly, in some embodiments, the requesting of data (to be sent to the receiving device) may be delayed until a quantity of storage space of PPS size is available for receiving. Then, when at least a PPS quantity of space is available, the requesting of data of PPS size may be made. In some embodiments, where the peripheral-to-peripheral link is PCIe the PPS may be, for example, set to 4096 bytes. In some embodiments the PPS may be set to, for example, 2Nbytes, where N is an integer so that, for example the PPS may be set to 4 bytes, 64 bytes or 2048 bytes, etc.

In some embodiments, a mechanism referred to as an “Eviction Timer” (ET) may be used in association with PPS. For example, an ET may be used to cause a data to be sent when less than a PPS quantity of data has been accumulated. For example, an ET may set a given period of time to wait for a given amount of data (e.g., a PPS) to accumulate. If enough data is accumulated (e.g., at least a PPS) before the expiration of the ET, then the data may be sent. If enough data is not accumulated before the expiration of the ET, then the data present may be sent, regardless of whether or not it meets the given amount. This may be useful when quantity of data waiting to be transferred is not a multiple of the PPS. An ET may be programmable and may be set, for example, to a value of 1 uS. Circuitry associated with an ET may reset the ET every time a new data item for sending is obtained.

In conjunction with sending, the accumulated transfer credit value may be adjusted, as depicted at block908. In one embodiment, the accumulated transfer credit value may be decremented by reducing the current value of transfer credits by a number indicative of the quantity of data sent. In another embodiment, the accumulated transfer credit value may be decremented by increasing a number indicative of the value of used transfer credits. Other embodiments may utilize various other methods of counting the amount of credits used and comparing the used credits against the number of credits granted, thereby decreasing (as data is sent) accumulated transfer credit value.

After transferring one or more data items, the source device may determine if there is more data to send as depicted at block910. If there are more data items to be sent then while the current value of transfer credits meets/exceeds a threshold, as determined at block904, the transferring of data may continue. If however, the current value of transfer does not meet/exceed a threshold, the source device may wait for additional transfer credits to be granted (e.g., looping at block904). In one embodiment the threshold may be set to zero but in other embodiments the threshold may be non-zero. The threshold may be set so that, for example, a specific type of data transfer may not cause the value of granted transfer credits to fall below zero.

FIG. 10is a flowchart that depicts an exemplary method1000for receiving data in accordance with one or more embodiments of the present technique as may be performed by a target (e.g. receiving) peripheral. In the illustrated embodiment, method1000may include determining whether there is data to receive, as depicted at block1002. In one embodiment, a target peripheral may determine if there is data to be received. This may involve determining if there are initial data to be received and it may also involve determining if there are new data to be received. On determining that there is no new (or initial) data to be received the target device may wait (e.g., perform other activities) until it is determined that there is data to receive. In some embodiments, determining if there is data to receive may involve determining if there is sufficient storage space to receive data.

In one embodiment, method1000may include determining whether or not to grant transfer credits, as depicted at block1004. For example, before starting to receive data, the receiving device may determine if it should grant transfer credits to the sending device. As discussed above, if the sending device does not have sufficient transfer credits it may not send data. Determining whether to grant transfer credits may involve many factors. For example, it may involve determining the value of transfer credits already granted, the quantity of data transmitted, and/or determining free storage space on the receiving device.

In some embodiments, method1000may include determining the value of transfer credits to grant, as depicted at block1006. Determining this value may involve many factors. For example, it may involve determining the value of transfer credits already granted, the quantities of data transmitted, and/or free storage space. Once the value of transfer credits to grant has been determined, the credits may be granted, as depicted at block1008. In one embodiment, credits are granted/issued to the source device. Granting of the transfer credits may enable the source peripheral to start or continue sending data items as described above. As depicted at block1010, method1000may also include receiving one or more data items. For example, after sufficient credits have been granted, the source device may send data as described above, and the data may be received at the target device. As depicted, method1000may also include returning to block1002to determine if there is more data to be received. As described above, the process of sending and receiving described with respect toFIGS. 9 and 10may occur in concert (e.g., in parallel) such that data may be streamed continuously between the devices.

Although the embodiments above have been described in considerable detail, numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications. Furthermore, note that the word “may” is used throughout this application in a permissive sense (e.g., having the potential to, being able to), not a mandatory sense (e.g., must). The term “include”, and derivations thereof, mean “including, but not limited to”. As used in this specification, the singular forms “a”, “an” and “the” include plural referents unless the content clearly indicates otherwise. Thus, for example, reference to “a device” includes a combination of two or more devices.