Source: http://patents.com/us-9973752.html
Timestamp: 2019-01-23 17:42:36
Document Index: 596205106

Matched Legal Cases: ['application No. 11878478', 'application No. 201180075963', 'application No. 11878478', 'application No. 11878478', 'application No. 201180075963', 'application No. 201180075963']

US Patent # 9,973,752. Intelligent MSI-X interrupts for video analytics and encoding - Patents.com
United States Patent 9,973,752
Doddapuneni , et al. May 15, 2018
Doddapuneni; Naveen (Phoenix, AZ), Mishra; Animesh (Pleasanton, CA), Rodriguez; Jose M. (San Jose, CA)
Family ID: 1000003296348
13/994,822
PCT/US2011/067448
WO2013/100919
US 20140294102 A1 Oct 2, 2014
Current CPC Class: H04N 19/20 (20141101); H04N 19/42 (20141101); H04N 19/137 (20141101); H04N 19/87 (20141101); H04N 19/436 (20141101)
Current International Class: G06F 13/24 (20060101); H04N 19/436 (20140101); H04N 19/20 (20140101); H04N 19/42 (20140101); H04N 19/87 (20140101); H04N 19/137 (20140101)
6970206 November 2005 Swan
7725637 May 2010 Kumar
8793188 July 2014 Larkin
2005/0249483 November 2005 Bramley
2006/0013123 January 2006 Ryu
2006/0150071 July 2006 Chen
2006/0294277 December 2006 Tetrick
2008/0162762 July 2008 Neiger
2008/0276027 November 2008 Hagita
2011/0029101 February 2011 Castro Scorsi et al.
2011/0066783 March 2011 Konireddygari
2011/0093637 April 2011 Gupta
2011/0153893 June 2011 Foong
2011/0246686 October 2011 Cavanagh, Jr.
1391827 Feb 2004 EP
"Interrupts Delivery in a Multi-host Environment" by, Cheng-Chun Tu, Sep. 25, 2012, Retrieved from http://www3.cs.stonybrook.edu/.about.live3/files/pcie-interrupt-delivery.- pdf. cited by examiner .
"Xilinx PCI Express Interrupt Debugging Guide" by Xilinx, Jan. 15, 2014, retrieved from http://www.xillinx.com/Attachment/Xilinx_Answer_58495_PCle_Interrupt_Debu- gging_Guide.pdf. cited by examiner .
Office Action dated Aug. 5, 2014 by the European Patent Office in application No. 11878478.4 (3 pages). cited by applicant .
PCT International Search Report and Written Opinion of the International Searching Authority issued in corresponding PCT/US2011/067448 dated Aug. 27, 2012 ( 9 pages). cited by applicant .
PCT International Search Report and Written Opinion issued in corresponding PCT/US2011/067448 dated Aug. 27, 2012 (9 pages). cited by applicant .
CN office action in corresponding CN application No. 201180075963.2 dated Feb. 4, 2017. cited by applicant .
EP office action in corresponding EP application No. 11878478.4 dated Jul. 22, 2015. cited by applicant .
EP office action in corresponding EP application No. 11878478.4 dated Oct. 13, 2016. cited by applicant .
CN office action in corresponding CN application No. 201180075963.2 dated Sep. 26, 2017 (4 pages). cited by applicant .
CN office action in corresponding CN patent application No. 201180075963.2 dated Mar. 20, 2018 (4 pages) [no English translation. cited by applicant.
1. A method for encoding multiple video streams in parallel comprising: generating a message signaled interrupt, from a video analytics engine to a host, that indicates an interrupt; providing information as part of the interrupt including address and data values for the interrupt to assist in servicing the interrupt; and sending a single message to the host including both the interrupt and the information.
2. The method of claim 1 including providing a traffic class for the interrupt.
3. The method of claim 1 including holding off an acknowledge to the interrupt until data transfer to the host is complete.
4. A non-transitory computer readable medium storing instructions executed by a processor to perform a method for encoding multiple video streams in parallel comprising: generating a message signaled interrupt, from a video analytics engine to a host, that indicates an interrupt; providing information as part of the interrupt to assist in servicing the interrupt including address and data values for the interrupt; and sending a single message to the host including both the interrupt and the information.
5. The medium of claim 4 further storing instructions executed to perform a method including providing a traffic class for the interrupt.
6. The medium of claim 4 further storing instructions executed to perform a method including holding off an acknowledge to the interrupt until data transfer to the host is complete.
7. An apparatus for encoding multiple video streams in parallel comprising: a video analytics engine; and an interrupt controller to generate a message signaled interrupt, from the video analytics engine to a host, that indicates an interrupt, send a single message to the host including both the interrupt and the information, and said interrupt itself including information as part of the interrupt including address and data values for the interrupt to assist in servicing the interrupt.
8. The apparatus of claim 7, said controller to provide a traffic class for the interrupt.
9. The apparatus of claim 7, said controller to hold off an acknowledge to the interrupt until data transfer to the host is complete.
A video analytics engine 20 may be coupled to the host via a bus 18. In one embodiment, the video analytics engine may be a single integrated circuit which provides both encoding and video analytics. In one embodiment, the integrated circuit may use embedded Dynamic Random Access Memory (EDRAM) technology. However, in some embodiments, either encoding or video analytics may be dispensed with. In addition, in some embodiments, the video analytics engine 20 may include a memory controller that controls an on-board integrated two dimensional matrix memory, as well as providing communications with an external memory.
Thus, in the embodiment illustrated in FIG. 1, the video analytics engine 20 communicates with a local dynamic random access memory (DRAM) 19. Specifically, the video analytics engine 20 may include a memory controller 50 for accessing the memory 19. Alternatively, the video analytics engine 20 may use the system memory 22 and may include a direct connection to system memory.
The video capture interface automatically captures and copies each input frame. One copy of the input frame is provided to the VAFF unit 66 and the other copy may be provided to VEFF unit 68. The VEFF unit 68 is responsible for storing the video on the external memory, such as the memory 22, shown in FIG. 1. The external memory may be coupled to an on-chip system memory controller/arbiter 51 in one embodiment. In some embodiments, the storage on the external memory may be for purposes of video encoding. Specifically, if one copy is stored on the external memory, it can be accessed by the video encoders 32 for encoding the information in a desired format. In some embodiments, a plurality of formats are available and the system may select a particular encoding format that is most desirable.
In one embodiment, a time multiplexing of each of the plurality of streams may be undertaken in each of the video encoders 32 and the video analytics functional unit 42. For example, based on user input, one or more frames from the first stream may be encoded, followed by one or more frames from the second stream, followed by one or more streams from the next stream, and so on. Similarly, time multiplexing may be used in the video analytics functional unit 42 in the same way wherein, based on user inputs, one or more frames from one stream are subjected to video analytics, then one or more frames from the next stream, and so on. Thus, a series of streams can be processed at substantially the same time, that is, in one shot, in the video encoders and video analytics functional unit.
In some embodiments, the user can set the sequence of which stream is processed first and how many frames of each stream are processed at any particular time. In the case of the video encoders and the video analytics engine, as the frames are processed, they can be output over the PCI Express bus 36.
The context of each stream in the video encoder may be retained in a register dedicated to that stream in the register set 122, which may include registers for each of the streams. The register set 122 may record the characteristics of the encoding which have been specified in one of a variety of ways, including a user input. For example, the resolution, compression rate, and the type of encoding that is desired for each stream can be recorded. Then, as the time multiplexed encoding occurs, the video encoder can access the correct characteristics for the current stream being processed from the register 116, for the correct stream.
Similarly, the same thing can be done in the video analytics functional unit 42 using the register set 124. In other words, the characteristics of the video analytics processing or the encoding per stream can be recorded within the registers 124 and 122 with one register reserved for each stream in each set of registers.
Thus, referring to FIG. 6, the sequence 130 may be implemented in software, firmware, and/or hardware. In software or firmware based embodiments, the sequence may be implemented by computer executed instructions stored in a non-transitory computer readable medium, such as an optical, magnetic, or semiconductor memory. For example, in the case of the video encoder 32, the sequence may be stored in a memory within the encoder and, in the case of the analytics functional unit, they may be stored, for example in the pixel pipeline unit 44, in one embodiment.
If a processing change has been received, it may be stored in the appropriate shadow registers 114 or 116, as indicated in block 140. Then, when a current processing task is completed, the change can be automatically implemented in the next set of operations, be it encoding, in the case of video encoders 32 or analytics, in the case of the video analytics functional unit 42.
In some embodiments, the frequency of encoding may change with the magnitude of the load on the video encoder. Generally, the encoder runs fast enough that it can complete encoding of one frame before the next frame is read out of the memory. In many cases, the encoding engine may be run at a faster speed than needed to encode one frame or set of frames before the next frame or set of frames has run out of memory.
The context registers may store any necessary criteria for doing the encoding or analytics including, in the case of the video encoder, resolution, encoding type, and rate of compression. Generally, the processing may be done in a round robin fashion proceeding from one stream or channel to the next. The encoded data is then output to the Peripheral Components Interconnect (PCI) Express bus 36, in one embodiment. In some cases, buffers associated with the PCI Express bus may receive the encoding from each channel. Namely, in some embodiments, a buffer may be provided for each video channel in association with the PCI Express bus. Each channel buffer may be emptied to the bus controlled by an arbiter associated with the PCI Express bus. In some embodiments, the way that the arbiter empties each channel to the bus may be subject to user inputs.
Thus, referring to FIG. 3, a system for video capture may be implemented in hardware, software, and/or firmware. Hardware embodiments may be advantageous, in some cases, because they may be capable of greater speeds.
Initially, a check at diamond 82 determines whether a store command has been received. Conventionally, such commands may be received from the host system and, particularly, from its central processing unit 12. Those commands may be received by a dispatch unit 34, which then provides the commands to the appropriate units of the video analytics engine 20, used to implement the command. When the command has been implemented, in some embodiments, the dispatch unit reports back to the host system.
If a store command is involved, as determined in diamond 82, an initial memory location and two dimensional size information may be received, as indicated in block 84. Then the information is stored in an appropriate two dimensional matrix, as indicated in block 86. The initial location may, for example, define the upper left corner of the matrix. The store operation may automatically find a matrix within the memory 28 of the needed size in order to implement the operation. Once the initial point in the memory is provided, the operation may automatically store the succeeding parts of the matrix without requiring additional address computations, in some embodiments.
Referring back to FIG. 2, the video analytics functional unit 42 may be coupled to the rest of the system through a pixel pipeline unit 44. The unit 44 may include a state machine that executes commands from the dispatch unit 34. Typically, these commands originate at the host and are implemented by the dispatch unit. A variety of different analytics units may be included based on application. In one embodiment, a convolve unit 46 may be included for automated provision of convolutions.
Other video analytics units, in some embodiments, may include a centroid unit 48 that calculates centroids in an automated fashion, a histogram unit 51 that determines histograms in automated fashion, and a dilate/erode unit 52.
The Memory Transfer of Matrix (MTOM) unit 54 is responsible for implementing move instructions, as described previously. In some embodiments, an arithmetic unit 56 and a Boolean unit 58 may be provided. Even though these same units may be available in connection with a central processing unit or an already existent coprocessor, it may be advantageous to have them onboard the video analytics engine 20, since their presence on-chip may reduce the need for numerous data transfer operations from the video analytics engine 20 to the host and back. Moreover, by having them onboard the video analytics engine 20, the two dimensional matrix or main memory may be used in some embodiments.
Then, as indicated in block 110, regions with motion in excess of a threshold may be located. Only those regions may be encoded, in one embodiment, as indicated in block 112. In some cases, no regions on a given frame may be encoded at all and this result may simply be recorded so that the frame can be simply replicated during decoding. In general, the video encoder provides information in a header or other location about what frames were encoded and whether frames have only portions that are encoded. The address of the encoded portion may be provided in the form of an initial point and a matrix size in some embodiments.
As shown in FIG. 1, the video analytics engine 20 is coupled to a host including the central processing unit 12. The video analytics engine 20 executes instructions independently from the host central processing unit 12. However, the host central processing unit must feed the video analytics engine 20 both data and instructions and it must receive results of operations. To accomplish these tasks, without the overhead incurred in polling for completion of instruction execution, intelligent message signaled interrupts (MSI-X) may be applied in some embodiments.
To ensure data integrity for instructions that require a large data transfer to the host, the video analytics engine 20 uses a RAISE instruction that generates an MSI-X interrupt. The MSI interrupt that results not only serves as an interrupt but also carries additional information in the message data field of the interrupt to reduce the overhead involved in servicing the interrupt. Furthermore, the intelligent MSI-X interrupt controller holds off the acknowledge to the RAISE interrupt request from the instruction dispatch unit until the data transferred to the host is complete. This mechanism may ensure that an interrupt for a RAISE instruction is sent only after a successful completion of the READ or RMD instruction through the Peripheral Component Interconnect Express bus 36.
The structure of the MSI-X interface is as follows in one embodiment where IC is the video analytics engine 20, O is Out and I is In and size is in bytes.
TABLE-US-00001 Direc- tion w.r.t. Port Size IC Description misx_addr 64 0 The address value for the MSI-X misx_data 32 0 The data value for the MSI-X msi_req 1 0 Request from the application to send an MSI when MSI is enabled. Once asserted, msi_req must remain asserted until the EPC asserts msi_grant mis_tc 3 0 Traffic Class of the MSI(-X) request, valid when msi_req is asserted mis_grant 1 I One-cycle pulse that indicates that the EPChas accepted the request to send an MSI(-X). After asserting msi_grant for one cycle, the EPC does not wait for msi_req to be deas- serted then reasserted to generate another MSI. If msi_req remains asserted after the EPC asserts msi_grant for one cycle, the EPC will generate an- other MSI. Cfg_msix_en 1 I The MSI-X Enable bit of the MSI-X Control register in the MSI-X Capability structure Cfg_msix_func_mask 1 I The IC Mask bit of the MSI-X Control register in the MSI-X Capability structure
Referring to FIG. 7, the interrupt controller 300 receives clocks from the various components that provide interrupts and receives reset signals from those same devices. A configuration and status register (CSR) decode 302 receives CSR inputs. It provides a signal to MSI-X interface 304. It also provides a decode signal to the legacy interrupt pending register 306. The MSI-X interface receives interrupts from a resync unit 310. The resync unit 310 receives interrupts from functional units such as a video encoder (VE) the memory matrix (MM), the video capture interface (VCI), the external memory (DDR), the I.sup.2c bus (I2C), the general purpose input/output (GPIO), the dispatch unit (DU) and receives the dispatch unit RAISE signal.
Previous Patent US 9,973,751 | Next Patent US 9,973,753