Method and apparatus for overlaying a displayable image with a second image

The overlaying of video image data from an external source, such as a video camera, onto a representation of display data becomes accomplished by using a serial port of an auxiliary frame buffer to receive the serial data from the external source. The source data then becomes transferred from serial registers included within the auxiliary frame buffer into the randomly accessible array of the auxiliary frame buffer. Subsequent execution of a DMA (Direct Memory Access) places the source data overlaid into the image data in the master frame buffer for communication to the video display. Separate designation of the memory address in the auxiliary frame buffer and the main frame buffer allow the overlaying of data from the auxiliary frame buffer into the main frame buffer asynchronously or in a manner not requiring synchronization of the displayed positions relative to one another.

This invention is in the field of data processing systems, specifically as 
directed to such systems which visually display digital information. 
BACKGROUND OF THE INVENTION 
Modern data processing systems generally provide visual output by way of 
such output devices as cathode-ray tubes (CRTs). The performance of such 
systems can be enhanced when the visual display can receive information 
from more than one source, and which can overlay the information received 
from a first source (background) with the information received from a 
second source (foreground). The overlaying of such information can 
especially be useful where only a portion of the background information is 
hidden by the foreground information, as where the foreground information 
occupies a "window" of the display, and where the information hidden by 
the foreground window is not irretrievably lost. 
It is especially useful in such systems to overlay the output of a data 
processing system with a dissimilar source, such as a digitized signal 
from a video camera. In such a system using a real-time source such as a 
video camera, it is preferable that the overlaying and display of the 
foreground information occur on a real-time basis, in order to take full 
advantage of the video camera input. 
Referring to FIGS. 1a and 1b, prior methods for displaying images from 
multiple sources are illustrated. FIG. 1a illustrates the overlaying of 
information from multiple digital sources 2. The output of each digital 
source 2 can represent the video data in the well-known RGB 
(red-green-blue) format. Digital-to-analog converters 4 receive the RGB 
information from each of digital sources 2, and convert the digital RGB 
information into analog signals for each of the elements therein. The 
output of the DACs 4 is then received by an analog mixer 6, which selects 
among the signals from the multiple DACs 4 as required for the overlaying 
of information among the digital sources 2. FIG. 1b illustrates a second 
system, containing frame buffers 8 which store bit-mapped representations 
of the information to be displayed on the video output device. Each of 
frame buffers 8 communicate with a data source (not shown), such as a host 
processing unit or the digitized output of a video camera, by way of its 
parallel port 10; each of frame buffers 8 also have, as is well known in 
the art, a serial port 12 for communicating the bit-mapped data to the 
video display. Serial ports 12 are received by a digital mixer 14, which 
constitutes a multiplexing function (or another operation on the serial 
digital data received, such as a logical OR) on each of the digital values 
presented by serial ports 12 thereto. The output of digital mixer 12 is 
applied to a DAC 16, for providing the analog RGB output as in FIG. 1a 
for application to a video display device. 
A problem in each of the prior methods shown in FIGS. 1a and 1b is the 
synchronization of data from the multiple sources. The data presented to 
each of the mixers (analog mixer 6 of FIG. 1a, and digital mixer 14 of 
FIG. 1b) from each of the sources must correspond to the same position on 
the video display, thereby requiring that the data from the foreground 
source must be presented at the same rate as, and synchronized with, the 
data from the background source. This places constraints on the data 
arriving from the foreground source, as the video display position of the 
foreground image must coincide with the position of the hidden portion of 
the background image. This requires that the foreground image be buffered, 
and thereby delayed, to coincide with the operation of the video display 
as controlled for the background image. 
It is therefore an object of this invention to provide a system for 
overlaying a foreground image onto a background image, without requiring 
that the display positions of the two images be synchronized. 
It is another object of this invention to provide such a system which 
allows the foreground image to be input from a real-time source, such as a 
video camera. 
It is yet another object of this invention to provide such a system which 
allows the overlaying of the foreground image in selectable positions of 
the background image, such selection being transparent to the receipt and 
transmission of the foreground image. 
Other objects and advantages of the instant invention will be apparent to 
those of ordinary skill in the art having reference to the following 
specification in conjunction with the drawings. 
SUMMARY OF THE INVENTION 
The invention may be incorporated into a video display system having a 
first frame buffer for storing a displayable image and for communicating 
the stored image to a video output device, and having a second frame 
buffer for receiving data representing a foreground image to be overlayed 
onto the image stored in the first frame buffer. The second frame buffer 
has a serial input for receiving the foreground image data, and has a 
random access output connected to a data bus; the random access input of 
the first frame buffer is also connected to the data bus. The second frame 
buffer is operable to receive and store the foreground image 
asynchronously from the operation of the first frame buffer. At such time 
as the foreground image is to be transferred to the first frame buffer, a 
direct-memory-access (DMA) transaction takes place over the data bus. An 
address latch may be provided to the second frame buffer, loadable from 
the data bus prior to the DMA operation, for storing the memory locations 
of the second frame buffer to be read in the DMA operation; a similar 
latch may be provided to the first frame buffer for storing the memory 
locations of the first frame buffer to which the data is to be written in 
the DMA operation. The independent control of the addresses allow the 
second frame buffer to receive, store and transfer the image without being 
synchronized with the video display position of the first frame buffer. 
Additional interface may be provided to each of the frame buffers for 
image processing on the data stored therein, as desired, such image 
processing capable of being done asynchronously relative to the other 
frame buffer.

DETAILED DESCRIPTION OF THE INVENTION 
FIG. 2 illustrates, in block form, a frame buffer 20 as used in the system 
to be described below. The frame buffer 20 of FIG. 2 has a serial port 22, 
and a parallel port 24, each of which may receive and present data from 
video RAMs (VRAMs) 26 comprising frame buffer 20. VRAMs 26 are 64k by 4 
dual-port dynamic random access memories, such as the TMS 4461 video RAM 
manufactured and sold by Texas Instruments Incorporated, and such as the 
device described in U.S. Pat. No. 4,636,986 issued Jan. 13, 1987 and 
assigned to Texas Instruments Incorporated. VRAM 26 has four random access 
input/output terminals DQ, and has four serial input/output terminals SDQ, 
allowing random and serial access to occur independently and 
asynchronously relative to one another. As described in said U.S. Pat. No. 
4,636,986, each VRAM 26 can transfer data from a selected row of memory 
data in the random access array to a register, for the serial output from 
serial port 22, and can receive serial data from serial port 22 into the 
register, for transfer into a row of the random access array therein. 
Serial port 22 of frame buffer 20 constitutes the four serial input/output 
terminals SDQ from the eight VRAMs 26 therein, providing for the 
communication of thirty-two bits of data in the form of thirty-two 
parallel serial data streams. The serial receive or transmit function is 
controlled for VRAMs 26 by a serial clock signal on line SCLK. Parallel 
port 24 of frame buffer 20 constitutes the four random access terminals DQ 
from the eight VRAMs 26 therein, providing for the parallel communication 
of thirty-two bits of data for each random access of VRAMs 26 therein. 
Address bus 28 carries address signals to all eight of VRAMs 26 in frame 
buffer 20, and control bus 30 carries the necessary clock and control 
signals (such as RAS.sub.--, CAS.sub.--, WE.sub.--, TR.sub.--, etc.) to 
all eight of VRAMs in frame buffer 20. 
It should be noted that frame buffer 20 may be comprised of standard (i.e., 
single random access port) random access memories which provide output to 
and receive input from discrete shift registers external thereto, in order 
to accomplish the frame buffering functions. It will become apparent, 
however, that the performance of dual-port video RAMs makes their use 
preferable in the system constructed according to the instant invention. 
In addition, other types of video RAMs, such as the TMS 4161 (a 64k by 1 
dual port memory), or such as the device described in U.S. Pat. No. 
4,639,890 issued Jan. 27, 1987 and assigned to Texas Instruments 
Incorporated, may be utilized in frame buffer 20. It should further be 
noted that alternate configuration of VRAMs 26 (such as multiple banks of 
VRAMs 26, each bank selectable by decoded clock signals on serial clock 
line SCLK and on control bus 30, as is well known in the art) may be 
incorporated into frame buffer 20 depending upon the size of the 
bit-mapped display to be stored therein. 
Referring now to FIG. 3, a video system constructed according to the 
invention will be described. Host interface 32 is logic for communicating 
between the video system and a host computer 33, such as a personal 
computer. Bus 34 is a data, address and control bus to which such a 
personal computer may be connected. Host interface 32 controls the 
communication from bus 34 to data bus 36, address bus 38 and control bus 
40. Parallel port 24.sub.m of main frame buffer 20.sub.m is connected to 
data bus 36, for receiving data therefrom and transmitting data thereto. 
Data bus 36 is further connected to row latch 42.sub.m and column latch 
44.sub.m, for communicating the start address of a memory transfer as will 
be described herein. Address bus 38 is connected to tristate buffer 
46.sub.m. The outputs of row latch 42.sub.m, column latch 44.sub.m and 
tristate buffer 46.sub.m are connected together and communicated to the 
address inputs of video system controller 48.sub.m. Video system 
controller (VSC) 48.sub.m is an integrated circuit for controlling the 
operation of frame buffer 20.sub.m, and is of the type such as the TMS 
34061, manufactured and sold by Texas Instruments Incorporated. VSC 
48.sub.m controls the operation of frame buffer 20.sub.m by the 
application of address signals thereto via address bus 28.sub.m, and by 
the application of control signals via control bus 30.sub.m. As is well 
known for the TMS 34061, VSC 48.sub.m can, for example, control frame 
buffer 20.sub.m to the extent that intervention of the host computer is 
not necessary for the refresh of the video display driven thereby. Control 
logic 50.sub.m is connected to control bus 40 from host interface 32, and 
controls the operation of row latch 42.sub.m, column latch 44.sub.m, 
tristate latch 46.sub.m, and VSC 48.sub.m according to signals on control 
bus 40; control logic 50.sub.m can consist of a programmable logic array, 
or a microcomputer, depending upon the application, as is well known in 
the art for controlling a video display. The control lines from control 
logic 50.sub.m to the various other constituents of the display system on 
the side associated with frame buffer 20.sub.m are not shown, for purposes 
of clarity. 
Serial port 22.sub.m of frame buffer 20.sub.m communicates the output of 
VRAMs 26 contained therein to latch/DAC circuitry 52. Latch/DAC circuitry 
52 is such logic as is well known in the art for the formatting of 
thirty-two bits of serial data into RGB signals for driving a display 54. 
Latch/DAC circuitry 52 provides the serial clock signal on line SCLK to 
control the timed serial output of video data from frame buffer 20.sub.m 
as needed for the display 54. Accordingly, VSC 48.sub.m, under the control 
of control logic 50.sub.m with addresses driven by way of address bus 38 
through tristate latch 46.sub.m, can receive data from the host computer 
33 through host interface 32, and control the writing of this data to 
frame buffer 20.sub.m and eventually to display 54. Once frame buffer 
20.sub.m stores a bit-mapped representation of the image to be displayed, 
VSC 48.sub.m controls refresh of the image on display 54, and updating of 
the displayed image as communicated from host interface 32. 
In the video system according to the invention, an auxiliary frame buffer 
20.sub.a is provided for receipt of timed digital image or source data 
information from a timed video source 56, such as a video camera. Video 
source 56 communicates data to NTSC/RGB decoder and formatter 58. NTSC/RGB 
decoder/formatter 58 consists of logic for translating timed source data 
from the format provided by video source 56 (e.g., data according to the 
NTSC standard), into thirty-two serial data streams of RGB data, in a 
format suitable for receipt and storage by frame buffer 20.sub.a, through 
serial port 22. NTSC/RGB logic 58 further generates the serial clock 
signal on line SCLK for controlling the storing of the serial data into 
VRAMs 28 within frame buffer 20.sub.a. Similarly as for frame buffer 
20.sub.m, frame buffer 20.sub.a has associated therewith VSC 48.sub.a, row 
address latch 42.sub.a, column address latch 44.sub.a, tristate buffer 
46.sub.a, and control logic 50.sub.a. Tristate buffer 46.sub.a receives an 
address signal via address bus 38 from host interface 32, while row and 
column address latches 42.sub.a and 44.sub.a, respectively, store values 
presented thereto from data bus 36. 
Control logic 50.sub.a is for controlling the operation of VSC 48.sub.a, 
row address latch 42.sub.a, column address latch 44.sub.a, and tristate 
buffer 46.sub.a, responsive to control bus 40 from host interface 32, in a 
manner similar as control logic 50.sub.m. In addition, control logic 
50.sub.a communicates with control logic 50.sub.m in a "handshaking" 
manner to effect the transfer of data from frame buffer 20.sub.a to frame 
buffer 20.sub.m, as will be described hereinbelow. Line XFR carries a 
signal from control logic 50.sub.m to control logic 50.sub.a to request a 
transfer of data from frame buffer 20.sub.a to frame buffer 20.sub.m, and 
line ACK carries a signal from control logic 50.sub.a to control logic 
50.sub.m indicating that such a transfer is ready to begin. Similarly as 
control logic 50.sub.m, control logic 50.sub.a may be a logic array; 
however, it is preferable from the standpoint of independent operation 
that control logic 50.sub.a be a microcomputer such as a TMS 7042 
manufactured and sold by Texas Instruments Incorporated, as certain amount 
of the work in effecting the transfer of data from frame buffer 20.sub.a 
to frame buffer 20.sub.m can be facilitated by such a device. Data 
terminals of control logic 50.sub.a may be connected, via address bus 51, 
to the inputs of row address latch 42.sub.a and column address latch 
44.sub.a, so that control logic 50.sub.a can establish and control, 
without the intervention of the portion of the system directed to the 
display, the addressing of the image to be overlayed. 
The system of FIG. 3 further includes such timing circuitry as necessary to 
effect the DMA operation of transferring the overlay image from frame 
buffer 20.sub.a to frame buffer 20.sub.m. Transfer clock 60 is operable 
responsive to control signals from control logic 50.sub.m, to generate the 
ALE signals to VSCs 48, and to increment the value of the row address 
latches 42 and column address latches 44. Transfer clock 60 may consist of 
such circuitry as to also effect the generation of the ALE signals in 
non-transfer modes, or it may be a redundant clock which is used solely 
for generating the ALE signals to VSCs 48 during the transfer cycles, with 
the associated control logic 50 controlling the timing of the operation of 
VSCs 48 in other operations. As is well known in the art concerning the 
operation of the TMS 34061, the ALE signal causes VSC 48 to latch the 
contents of its address and function select pins, and to effect the 
operative cycle designated thereby. As will be evident from the 
description below, the operation of the two VSCs in the transfer of image 
data from frame buffer 20.sub.a to frame buffer 20.sub.m will be 
substantially synchronous. 
The operation of the system of FIG. 3 in effecting the transfer of data 
from source 56, through frame buffer 20.sub.a to frame buffer 20.sub.m, 
and to display 54, accomplishing the overlay of the image stored in frame 
buffer 20.sub.m with the image stored in frame buffer 20.sub.a, will now 
be described. As described above, source 56 has communicated data through 
NTSC/RGB decoder/formatter 58 into the thirty-two serial registers within 
VRAMs 28 in frame buffer 20.sub.a. VSC 48.sub.a presents the control 
signals necessary to effect the register-to-memory transfers within VRAMs 
28 of frame buffer 20.sub.a, as well as the row address to which the 
transferred contents are transferred. This row address can be presented by 
address bus 38 via tristate latch 46.sub.a, and stored in one of the 
registers located within VSC 48.sub.a for application to frame buffer 
20.sub.a and updating by various operations performable by VSC 48.sub.a. 
This operation of the receipt and storing of the input data from source 56 
is performable independently from and asynchronously with the operation of 
frame buffer 20.sub.m, under the control of VSC 48.sub.m, in refreshing 
and updating the image displayed by display 54. 
Prior to effecting a transfer from frame buffer 20.sub.a to frame buffer 
20.sub.m, the address in frame buffer 20.sub.a from which the transfer is 
to begin is loaded into row address latch 42.sub.a and column address 
latch 44.sub.a from data bus 36, under the control of control logic 
50.sub.a ; the values of the row and column address to be loaded therein 
may be presented by data from host interface 32, or from control logic 
50.sub.a itself via bus 51 (if a microcomputer such as a TMS 7042 is used 
as control logic 50.sub.a). Tristate buffer 46.sub.a is placed in tristate 
condition by control logic 50.sub.a, so that address bus 38 will not 
interfere with the application of the start address by row and column 
address latches 42.sub.a and 44.sub.a to VSC 48.sub.a. 
Similarly, row and column address latches 42.sub.m and 44.sub.m are loaded 
with the beginning address to which the overlay image is to be stored in 
frame buffer 20.sub.m. The provision of separate address latches 42 and 44 
for the two frame buffers 20 allow the transfer of data, along the lines 
of a direct-memory-access operation, with independently specified 
destination and source address values. Accordingly, unlike the prior art 
systems discussed hereinabove relative to FIGS. 1a and 1b, the transfer of 
data from frame buffer 20.sub.a to frame buffer 20.sub.m does not require 
that the source location in frame buffer 20.sub.a be synchronized with a 
particular location within frame buffer 20.sub.m. Instead, a DMA transfer 
such as is well known in the art, can be effected to place the data from a 
randomly selected location within frame buffer 20.sub.a into a randomly 
selected location within frame buffer 20.sub.m. 
Once the respective row and column address latches 42 and 44 are loaded 
with the desired source (frame buffer 20.sub.a) and destination (frame 
buffer 20.sub.m) values, the transfer can begin. The transfer is initiated 
by control logic 50.sub.m issuing a signal on line XFR to control logic 
50.sub.a ; such a signal will generally occur when control logic 50.sub.m 
has sensed that a desired operation has stopped on the display side of the 
system of FIG. 3. Control logic 50.sub.a will issue a signal on line ACK 
when it is available to effect a transfer; the transfer may, of course, be 
granted priority above, or subject to, other operations performed within 
the system of FIG. 3. Upon receipt of the signal on line ACK, control 
logic 50.sub.m then activates transfer clock 58, which begins the access 
of each of frame buffers 20.sub.m and 20.sub.a via their respective VSCs 
48.sub.m and 48.sub.a, respectively. 
The DMA operation occurs by way of a read of a location in frame buffer 
20.sub.a, and a write operation (of the data read from frame buffer 
20.sub.a) to a location in frame buffer 20.sub.m. Accordingly, VSC 
48.sub.a will present an address (along with the necessary clock signals, 
and with a read signal) to frame buffer 20.sub.a according to the values 
presented by row address latch 42.sub.a and column address latch 44.sub.a 
; VSC 48.sub.m will be presenting an address corresponding to the contents 
of row address latch 42.sub.m and column address latch 44.sub.m to frame 
buffer 20.sub.m, along with the necessary clocks. Frame buffer 20.sub.a 
will, responsive to the address value, present the contents of the 
addressed location onto data bus 36 (thirty-two bits wide); thereafter, 
frame buffer 20.sub.m will receive the value on data bus 36 and write it 
into the address specified by VSC 48.sub.m. The write signal applied to 
frame buffer 20.sub.m is preferably delayed for a brief period of time 
after the expected access time of frame buffer 20.sub.a, so that the data 
read from frame buffer 20.sub.a is stable on data bus 36 prior to the 
assertion of the write command signal, minimizing the occurrence of 
erroneous write operations. Such delay may be controlled by VSC 48.sub.m, 
as is well known in the art. 
The bandwidth of the DMA transfer from frame buffer 20.sub.a to frame 
buffer 20.sub.m may be increased by the use of the well known page mode 
cycle, to access frame buffers 20.sub.a and 20.sub.m. The page mode cycle 
allows a multiple number of column address cycles to take place without 
reselecting a row within VRAMs 28 in the frame buffers 20. The cycle time 
for such a page mode operation thus shortens from 230 nsec for a full 
cycle to 120 nsec for a page mode cycle, for the TMS 4461 device. Since 
image data stored in a bit-mapped form generally consists of contiguous 
columns within a contiguous block of rows, the page mode cycle can 
generally be used to increase the speed of the transfer. 
Upon each cycle, transfer clock 58 increments the contents of the row and 
column address latches 42 and 44, so that the next cycle has the next 
successive address. Of course, if the page mode cycle is to be used, logic 
must be included within the system to increment the column address latches 
44 without incrementing the row address latches 42 (except upon the 
completion of a row access); such logic is easily included by one of 
ordinary skill in the art, as required for any DMA operation utilizing the 
page mode feature of dynamic RAMs. 
Upon completion of the desired number of transfer cycles, control logic 
50.sub.m will signal transfer clock 58 that no further cycles for such a 
transfer are to be performed. Control logic 50.sub.m will inactivate the 
signal on line XFR to control logic 50.sub.a, and control logic 50.sub.a 
will inactivate the signal on line ACK in response thereto. The system 
will then return to its normal operating mode, with each of VSCs 48 
controlling its associated frame buffer 20 as described above. 
Once the DMA operation is complete, the image data from buffer memory 
20.sub.a is stored at a known location within frame memory 20.sub.m. 
Generally the stored location of this data will be in a location of frame 
memory 20.sub.m which is not contained within the bit mapped area of the 
primary image. As is well known in the art, however, VSC 48.sub.m can 
select the location of the overlay image within frame buffer 20.sub.m for 
display at the location desired in the display. The storage of the overlay 
data not within the bit-mapped area allows frame memory 20.sub.m to retain 
the data in the first image which is to be overlayed (when overlay is 
desired) for display at a later time. In this way, the overlay image may 
be "panned" across the display 54, with the original image data 
reappearing when the overlay image is moved; the overlay data may also 
appear in a "window", as is well known in the art, which may be selected 
by the user of the host system in an interactive manner. 
Further modifications to the system of FIG. 3 may of course be included. 
For example, a separate host interface may be provided for frame buffer 
20.sub.a, similar to host interface 32, so that processing of the image 
received from source 56 may be done independently from the operation of 
frame buffer 20.sub.m in conjunction with VSC 48.sub.m and host interface 
32. In addition, multiple auxiliary frame buffers 20.sub.a may be 
provided, with control logic 50.sub.m decoding which of the multiple 
auxiliary frame buffers 20.sub.a the DMA operation is to provide the 
overlay data. 
According to the above description, the overlaying of video image data from 
an external source 56, such as a video camera, onto a bit-mapped 
representation of display data may be accomplished by using the serial 
port 22 of an auxiliary frame buffer 20.sub.a to receive the serial data 
from source 56, transferring the data from the serial registers included 
within such an auxiliary frame buffer 20.sub.a into the randomly 
accessible array therein, and by subsequently executing a DMA operation to 
place the overlay image data into the master frame buffer 20.sub.m for 
communication to the display 54. Separate designation of the memory 
addresses in auxiliary frame buffer 20.sub.a and main frame buffer 
20.sub.m allow the overlaying of data asynchronously or in a manner not 
requiring synchronization of the display positions relative to one 
another. 
Although the invention has been described with reference to an illustrative 
embodiment, it is to be understood that this description is by way of 
example only, and is not intended to be construed in a limiting sense. It 
is to be further understood that numerous changes in the details of the 
illustrative embodiment, and additional embodiments of the invention, will 
be apparent to, and may be made by, persons skilled in the art having 
reference to this description. In addition, it is to be further understood 
that those skilled in the art may readily substitute present and future 
equivalent components for those described herein, in order to achieve the 
same result as the illustrative embodiment. It is contemplated that such 
changes, substitutions and additional embodiments are within the spirit 
and scope of the invention as hereinafter claimed.