Efficient transmission of video and audio over slave FIFO interface

Systems and methods for translation and transmission of video and audio data over a first-in-first-out interface (FIFO) in a field programmable gate array (FPGA) are provided. The method includes receiving audio and video data including a number of video frames, each with a plurality of video lines separated by a line blanking interval. A first video line is translated and transmitted to a packet-based network through the FIFO in the FPGA while concurrently buffering the audio data in an audio buffer in the FPGA. Next, at least a portion of the audio data in the audio buffer is transmitted to the packet-based network through the FIFO during the line blanking interval separating the first video line from a second video line. Where video frames are separated by frame blanking intervals the method further includes transmitting through the FIFO any data remaining in the buffer after the preceding line blanking interval.

TECHNICAL FIELD

This disclosure relates generally to translation and transmission of video and audio data, and more particularly to systems and methods for translation and transmission of video and audio data over (FIFO) interface.

BACKGROUND

It is frequently desirable to connect an uncompressed video from a multi-media source, such as a video camera, video player, game console or streaming device, to a display or recording device via a packet-based network, such as a Universal Serial Bus.FIG.1Ais a block diagram of a conventional system100for translating and transmitting video and audio data from a multi-media source102to a Universal Serial Bus (USB104) or USB interface.

Referring toFIG.1Athe system100includes a Field Programmable Gate Array (FPGA106) and a USB controller108. The FPGA106is configured to receive video and associated audio data from the multi-media source102, and generally includes a high bandwidth parallel interface, such as a First-In-First-Out (FIFO) interface110to combine and transmit the video and audio data over a parallel or FIFO bus112. The USB controller108is configured to receive the parallel data from the FIFO bus112, and convert the data into USB data packets, which are then transmitted or coupled over the USB104to a USB compatible display or recording device.

The FPGA106further includes a video data decoder114configured to receive video data and frame valid signals from an image sensor116in the multi-media source102, an audio data decoder118to receive audio data from a microphone120in the multi-media source, a video buffer122coupling the video data decoder to the FIFO interface110, and an audio buffer124coupling the audio data decoder to the FIFO interface. Since the FPGA106must transfer video and audio data to the USB controller108using the single FIFO interface110, and both audio and video are real-time data, the video buffer122and audio buffer124are required to buffer this data for sequential, interleaved transmission at regular intervals.

FIG.1Bis a timing diagram of a video and audio data and signals from the multi-media source102and an output from the FIFO interface110of the system100ofFIG.1A. Referring toFIG.1Bit is seen, video data is transmitted from the multi-media source as individual video frames, each made up of a number of lines of image data. To maintain integrity of each video frame a frame valid signal is transmitted from the multi-media source to indicate the beginning and end of each video frame. The interval between the end of one video frame and the beginning of another is referred to as video frame blanking. Conventionally, this interval between video frames is used by the system100to transmit audio data through the FIFO interface110. Thus, the audio buffer124must store audio data until the video frame blanking interval. For example, video frame blanking occurs once in 33.3 milliseconds (ms) for a 30 frames-per-second (fps) video, so the audio buffer124must have sufficient buffer memory to store audio data for at least 33.3 ms.

Additionally, the video buffer122and the audio buffer124frequently need to buffer data for a number of video frames and the associated audio data to support any throughput variations arising from delays in the FIFO interface110and USB controller108, resulting in the need for even larger buffer memories. This is significant since the cost of FPGAs106including large embedded memory adds substantially to the cost of the system100.

FIG.1Cis a block diagram of another system100′ for translating and transmitting video and audio data from the multi-media source to the USB104including an FPGA106′ including an inter-integrated circuit sound (I2S) interface126and I2S serial bus128, to transmit audio data from the audio buffer124to the USB controller108. This approach eliminates the need to limit transmission of audio data to video frame blanking intervals. However, the need to buffer both audio and video data to support throughput variation remains, and the added I2S interface126as well as the need for a USB controller capable of supporting both a FIFO and I2S interfaces adds substantially to the cost and complexity of the system100′.

Accordingly, there is a need for a system and method for translating and transmitting video and audio data from a multi-media source to USB through an FPGA including a FIFO interface, without the need for large embedded memory. It is further desirable, that the system and method not require additional interfaces in the FPGA or a USB controller for separate transmission of audio data.

SUMMARY

Systems and methods for translation and transmission of video and audio data over a first-in-first-out (FIFO) interface are provided. Generally, the method includes receiving in a field programmable gate array (FPGA) audio data, and video data including a number of video frames, each with a plurality of video lines separated by a line breaking or blanking interval. A first of the plurality of video is translated and transmitted to a packet based network through a single FIFO interface in the FPGA while concurrently buffering the audio data in an audio buffer in the FPGA. Next, at least a portion of the audio data in the audio buffer is transmitted to the packet based network through the FIFO interface during the line blanking interval separating the first video line from a second video line. Where each video frame in the video data received is separated from a succeeding video frame by a frame blanking interval, and the method further includes translating and transmitting to the packet based network through the FIFO interface any audio data in the audio buffer not translated and transmitted during an immediately preceding line blanking interval during the frame blanking interval separating a first video frame from a second video frame.

In some embodiments, the system is a universal serial bus (USB) bridge including a FPGA with a single FIFO interface operable for translating and transmitting video and audio data from a HDMI source to a USB controller. The system further includes a universal serial bus (USB) controller coupled between the FPGA and a USB network. Generally, the FPGA further includes a video data decoder operable to decode video data from a multi-media source, an audio data decoder operable to decode audio data from the multi-media source, and an audio buffer coupled to the audio data decoder. The FIFO interface is operable to translate and transmit a first video line of the plurality of video lines to a USB network through the USB controller while concurrently buffering the audio data received in the audio buffer, and to translate and transmit at least a portion of the audio data in the audio buffer to the USB network during the line blanking interval separating the first video line from a second video line in the video frame.

DETAILED DESCRIPTION

A system and methods are provided for translating and transmitting video and audio data from a multi-media source to a Universal Serial Bus (USB) over a slave First-in-First-out (FIFO) Interface. The system and methods of the present disclosure are particularly useful for translating and transmitting video and audio data from a High-Definition Multimedia Interface (HDMI) source to a USB.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be evident, however, to one skilled in the art that the present invention can be practiced without these specific details. In other instances, well-known structures, and techniques are not shown in detail or are shown in block diagram form in order to avoid unnecessarily obscuring an understanding of this description.

Reference in the description to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places in the specification do not necessarily all refer to the same embodiment. The term to ‘couple’ as used herein can include both to directly electrically connect two or more components or elements and to indirectly connect through one or more intervening components.

The present disclosure describes a system and method for efficiently translating and transmitting audio and video data in an uncompressed format, such as High-Definition Multimedia Interface (HDMI) format, through a slave (FIFO) interface to a packet based network, such as a Universal Serial Bus (USB) network. Briefly, the method involves interleaving audio data over video data more periodically than in conventional systems and methods by transmitting buffered audio data during every video line breaking or blanking time or interval, rather than solely between each video frame.FIG.2is a block diagram illustrating a timing relationship of line blanking intervals and frame blanking intervals for a single frame of High-Definition Multimedia Interface HDMI video data. Referring, toFIG.2, each frame of video data or video frame202includes multiple lines of video data or video lines204, which are transmitted through a HDMI interface sequentially from top to bottom of the video frame and separated during transmission by line blanking intervals206. Each video frame202is further separated from a second, sequentially transmitted video frame202′ by a frame blanking interval208. The number of pixels (not shown) of imaging data in each video line204and the number video lines in each video frame202depend on a size and resolution of video being transmitted. The duration or time period of the line blanking intervals206and the frame blanking intervals208depend on a frame rate of the video being transmitted. For example, for a video with a typical full high definition (FHD) resolution, each video frame202includes 1080 video lines204of 1920 pixels each, sent at 30 frame per second (fps), the line blanking intervals206are about 3.78 microseconds (μs) and the frame blanking intervals208are about 1.33 milliseconds (ms).

FIG.3is a block diagram of a USB bridge circuit or system300operable to translate and transmit video and audio data from a multi-media source301including an image senor302and microphone303, such as a HDMI compatible camera, to a recording or display device304through a packet based network, such a USB306, by buffering and interleaving transmission of audio data during line blanking intervals between transmission of video lines. Referring toFIG.3, the system300generally includes a receiver, such as a HDMI receiver308coupled to the multi-media source301through an HDMI link310, a field programmable gate array (FPGA312) including a slave first-in-first-out (FIFO) interface314, and a master controller, such as universal serial bus (USB) controller316, coupled to the slave FIFO interface in the FPGA through a general parallel interface (GPIF318). Generally, the GPIF318includes a clock signal, a number of FIFO addresses, and slave select, slave write and parallel data signals, through which the FPGA312can control the slave FIFO interface314.

The FPGA312includes in addition to the slave FIFO interface314a video data decoder320and an audio data decoder322coupled to the HDMI receiver308. The video data decoder320is coupled to the slave FIFO interface314through a video buffer324, and is operable to receive image data and signals including frame valid and line valid signals from the multi-media source301through the HDMI receiver308, and to translate or decode the image data and signals to generate video data, which is then communicated or transmitted to the slave FIFO interface through the video buffer. The video buffer324is operable to store or buffer at least a portion of the video data to support delays in transmission through the slave FIFO interface314, USB controller316and/or the packet based network i.e., USB306. Thus, a memory size of the video buffer324is selected based on the video frame rate a number of pixels in each video line and on the number of video lines in each video frame. Generally, the video buffer324is sized to support a wide range of video frame rates, and variable frame rates, up to and including HD video at 60 fps. For example, in one embodiment the video buffer324includes sufficient embedded memory to provide uninterrupted stream of 30 frames of FHD video data, i.e., 1080 video lines of 1920 pixels. By variable frame rates it is meant that the frame rate at which the video received can change during the reception of a single uninterrupted video.

The audio data decoder322is also coupled to the slave FIFO interface314through an audio buffer326, and is operable to receive audio data, for example in the form of pulse-density modulation (PDM) data, from the multi-media source301through the HDMI receiver308, and to translate or decode the PDM data to generate audio data, and to communicate or transmit the audio data to the slave FIFO interface through the audio buffer. The audio buffer326is operable to store or buffer decoded audio data associated with at least one line of video data while the line of video data is being translated and transmitted. Thus, an embedded memory size of the audio buffer326is selected based on a quanta in bytes of audio data associated with each line of video data received and transmitted, which in turn can depend on a number of channels and a sampling frequency of the audio data, and a time required to decode and transmit a line of video data, also known as the line active portions or time. The size of the embedded memory of the audio buffer326may be further selected based of the number of bytes that can be transmitted during the line blanking interval between lines of video data. For example, where the video data being received is in FHD at 30 fps, the line active time for each line of video data is about 1 ms, and the line blanking interval is about 3.78 μs. Thus, where the audio data includes 2 channels at a sampling frequency of 48 kHz the size of the audio buffer326required to store or buffer the audio data associated with each line of video data, is a minimum of about 192 bytes or about 1.5 kbits. However, it will be understood that the embedded memory may be considerably larger to support delays in transmission through the slave FIFO interface314.

The slave FIFO interface314includes logic elements to execute an algorithm to interleave the video and audio data received from the video buffer324and audio buffer326, and to communicate or transmit the video and audio data to the USB controller316through the GPIF318.

An embodiment of the algorithm is for transmitting video and audio data while minimizing the required embedded memory size the audio buffer326is illustrated in the state machine diagram ofFIG.4. Referring toFIG.4, the algorithm400begins with determining if the video is active (video active402). That is whether or not video data has or is being decoded by the video data decoder320and is being communicated or transmitted through the video buffer324to the slave FIFO interface314. If the video is active, the slave FIFO interface314communicates or transmits the video data to the USB controller316over the GPIF318(send video data404). If the video is not active, a determination is made as if there is a frame blanking interval (frame blanking406). If there is a frame blanking interval, the slave FIFO interface314communicates or transmits any audio data stored or buffered in the audio buffer326to the USB controller316(send audio data408). If there is not a frame blanking interval, a determination is made as if there is a line blanking interval (line blanking410). If there is a frame blanking interval, the slave FIFO interface314communicates or transmits at least a portion of the audio data stored or buffered in the audio buffer326to the USB controller316over the GPIF318(send audio data408), and if not the slave FIFO interface is configured or reconfigured to check for video active402.

A method of operating the system ofFIG.3to translate and transmit video and audio data from a multi-media source to a recording or display device through a packet based network, while buffering and interleaving transmission of audio data during line blanking intervals between transmission of video lines will now be described with reference toFIGS.5and6.FIG.5is a timing diagram of a video and audio data and signals from the HDMI source and FIFO output from the USB bridge ofFIG.3.FIG.6is a flowchart illustrating a method for translating and transmitting video and audio data from an HDMI source to a USB over a FIFO Interface.

Referring toFIGS.5and6, the method begins with receiving in the FPGA from a multi-media source video data and audio data (step602). Generally, the video data contains both imaging data including a number of video frames, each with multiple video lines, a line valid signal502to indicate line blanking portions or intervals504separating line active portions506, and a frame valid signal508to indicate frame blanking intervals510separating each video frame from a subsequent video frame. The audio data can include a number of channels of audio encoded using, for example, PDM modulation and a PDM clock signal.

Next, video data512in a first video line of the multiple video lines in a first video frame is decoded or translated, and transmitted to a packet based network through a slave FIFO interface in the FPGA, while concurrently buffering in an audio buffer audio data received and decoded in an audio data decoder in the FPGA (step604).

At least a portion of the audio data514stored in the audio buffer is transmitted to the packet based network through the FIFO interface during the line blanking interval504separating the first video line from a second video line in the video frame (step606).

Generally, method further includes during the frame blanking interval510separating a first video frame from a second video frame transmitting to the packet based network through the FIFO interface any audio data514in the audio buffer not translated and transmitted during an immediately preceding line blanking interval504(step608).

Thus, systems for translation and transmission of video and audio data over a FIFO interface in a FPGA and methods of operating the same have been disclosed. Embodiments of the present invention have been described above with the aid of functional and schematic block diagrams illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.

The foregoing description of the specific embodiments will so fully reveal the general nature of the invention that others can, by applying knowledge within the skill of the art, readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of the present invention.