Abstract:
An apparatus for 2-D filtering a raster scanned image at high real-time rates such as those required for HDTV, with either IIR of FIR structures, without the need for expensive digital systems that require A/D and D/A converters. The wide variety of 2-D signal processing techniques (e.g. high pass filtering) that are based on a 2-D transfer function of general order can be implemented. The use of a semi-systolic structure allows a number of advantages for VLSI implementation such as modularity, regularity and high parallelism. Use of analog components in the processing elements of the apparatus makes high-speed operation for real-time applications in HDTV possible. The apparatus is applicable to TV imaging in general including such areas as advanced television systems, industrial and biomedical video systems.

Description:
BACKGROUND OF THE INVENTION 
     In the development of new receivers for television, more advanced signal processing techniques will be implemented in the circuitry of the video processing section. It has been proposed that such advanced video signal processing can be carried out by high-speed two-dimensional (2-D) spatial filtering or by three-dimensional (3-D) temporal filters. See for example, &#34;Digital Signal Processing in Television Receivers&#34; by M. J. J. C. Annergarn, A. H. H. J. Nillesen, and J. G. Raven, Philips Tech. Rev. 42, No. 6/7, 183-200, April 1986. Newer receivers will display pictures that have higher pixel resolution. For what has been termed High Definition Television (HDTV), if the processing is done in real-time, that is, at the same rate as the effective sampling rate of the picture, processing rates in excess of 80 million pixels/sec would be required (e.g. 1000 lines/field×50 fields/sec, 5:3 aspect ratio). 
     In the past analog signal processing techniques such as noise coring, edge peaking, and comb filter separation of luminance and chrominance signals have been based on one-dimensional time domain approaches implemented as simple Finite Impulse Response (FIR) signal processing structures. These have been quite limited as to the type of signal processing and enhancement operations that could be performed by the characteristic type of apparatus that resulted from following that approach. 
     More sophisticated signal processing is possible by means of the direct application of techniques for filtering 2-D data, known from the field of Mathematical Image Processing Theory. Various techniques for the enhancement of digital images are usually implemented by programming a general purpose computer to operate on stored digitized images off-line. These 2-D filtering techniques can be implemented in real-time by means of dedicated hardware. See for example, &#34;High-Speed Architectures for Digital Image Processing&#34;, by A. N. Venetsanopoulos, K. M. Ty, and A. C. P. Loui, IEEE Trans. on Circuits and Systems, Vol. CAS-34., No. 8, 887-895, August 1987. 
     Due to hardware constraints, conventional designs of dedicated 2-D filters have favoured FIR filters even though IIR (Infinite Impulse Response) filters are known to be more efficient allowing lower order realizations than their FIR counterparts. 
     Considerations of hardware complexity, physical size of apparatus, power consumption, and economical manufacture are all of vital importance in any practical signal processing apparatus intended for use in consumer products. 
     The present invention introduces a dedicated hardware apparatus for real-time 2-D filtering which is based on a semi-systolic analog structure realization. The hardware realization is based directly on the 2-D digital transfer function H(z 1 ,z 2 ) and utilizes off-the-shelf components. Both FIR and the more efficient IIR filtering structures can be realized with this approach and high real-time rates can be attained through the use of analog components that have inherently small delay times. Also since the structures are semi-systolic they have advantages for VLSI implementation such as modularity, regularity, and high parallelism. 
     Recently, motion adaptive digital filters have been proposed for use in high-definition television video signal processing. These can be regarded as common 3-D FIR filters, for the implementation of techniques for video signal processing. They require delays of one or more field periods, such delays being accomplished by means of frame-stores. Since pixels in separate fields are combined, this type of signal processing is referred to as temporal filtering and can only be performed on those pixels for which no motion in the scene of the picture being displayed has occurred between fields. Thus the development of such a filter is complicated by the necessary inclusion of circuitry that implements a motion detection algorithm. The embodiments of apparatus arising out of the present invention do not require analog to digital (A/D) and digital to analog (D/A) converters, often used in conjunction with analog pre-filters and post-filters, to convert video signal data from analog raster scanned form to sampled digital data for processing; nor do they require expensive frame-stores or motion detection circuitry. 
     Other methods of performing real-time 2-D signal processing have been based on elaborate algorithms such as the Burt Pyramid which separates an image into a number of 2-D spatial frequency bandpass images. See for example, &#34;A Two-Dimensional Real-Time Video Pyramid Processor&#34;, by J. H. Arbeiter and R. F. Bessler, RCA Review, Vol. 47, 3-31, March 1986. This method, although capable of real-time operation, has the disadvantage of greater complexity relative to the present invention, due to the need to process multiple bands and to generate a set of component images. When embodied in digital hardware, a large amount of circuitry is required along with the need for A/D, and D/A converters. 
     Theoretical aspects of the present invention have been described in &#34;Realization of 2-D IIR Filters using Sample-and-Hold Circuitry&#34;, by M. A. Sid-Ahmed, 1991 IEEE International Symposium on Circuits and Systems, Vol. 5, June 1991, pp. 2467-2470. 
     This application encompasses both theoretical as well as actual hardware implementation which is the combined work of both applicants. 
     SUMMARY OF THE INVENTION 
     The object of the present invention is to provide an apparatus for two-dimensionally filtering (i.e. signal processing) an image at high real-time rates such as those required for HDTV, with either IIR or FIR structures, without the need for expensive digital systems that require A/D and D/A converters. The wide variety of 2-D signal processing techniques (e.g. high pass filtering) that are based on a 2-D transfer function of general order can be implemented. 
     The use of a semi-systolic structure allows a number of advantages for VLSI implementation such as modularity, regularity and high parallelism. The apparatus of the present invention also has advantages in view of considerations of hardware complexity, physical size of apparatus, power consumption and economical manufacture. Use of analog components in the processing elements of the apparatus makes high-speed operation for real-time applications in HDTV possible. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 is a block diagram showing the partial realization of a 2×2 IIR embodiment of the invention; 
     FIG. 2 is a block diagram of a processing element (PE); 
     FIG. 3 is a block diagram of the form of the semi-systolic signal processing apparatus for the general case of an M×N IIR structure; 
     FIG. 4 is a block diagram of the form of the 2-D semi-systolic signal processing apparatus for the specific case of a 2×2 IIR structure; 
     FIG. 5 is a block diagram of a CCD 1H delay line; 
     FIG. 5B is a circuit diagram of the low pass filter circuit which was represented as a circuit block in the block diagram of FIG. 5. 
     FIG. 6 shows a practical circuit for a preferred embodiment of a processing element (PE). 
     FIG. 7 is a circuit diagram of a typical signal inverter. 
     FIG. 8A is a photograph showing two circuit boards, each board containing two line delays, used in a constructed prototype of a preferred embodiment of a 2×2 IIR filter apparatus; 
     FIG. 8B is a photograph showing a circuit board containing the circuitry, excluding line delays, of a constructed prototype of a preferred embodiment of a 2×2 IIR filter apparatus; 
     FIG. 9 shows a practical circuit for a preferred embodiment of a processing element (PE), in which the filter coefficients, which are the local variables of the PE, are digitally programmable. 
     FIG. 10 shows how the signal processing apparatus shown in the block diagram of FIG. 4 can be modified through the use of two types of processing element (PE) to allow the use of line delays which are all of standard length 1H; 
     FIG. 11 is a block diagram of a type II processing element (PE) as would be used to implement the invention as depicted in FIG. 10. 
     FIG. 12 is a block diagram showing how the 2-D semi-systolic signal processing apparatus is typically inserted into television video circuitry. 
    
    
     DETAILED DESCRIPTION 
     In general, a two-dimensional (abbreviated as 2-D) image can be described as a function of two independent spatial variables and time in the scene f(x,y,t). However, the time variable t is constant over a given image frame so for our purposes f(x,y,t) reduces to f(x,y). The image is converted to an electrical time varying signal for transmission by the process of periodic horizontal scanning. The resulting signal is often referred to as a raster scanned image. Interlacing is often introduced in TV signals to lower the video bandwidth requirements while simultaneously increasing the effective frame rate. In interlacing the even lines are scanned first, then all the odd lines, producing two alternate fields of lines for each picture frame. 
     In describing the present invention, we consider each field of the picture, whether interlaced or not, to be a separate image. A raster scanned image is interpreted as a 2-D input signal, consisting of individual horizontally scanned lines occurring in discrete time periods, each line being a continuous electrical time varying signal. 
     The input signal may optionally be sampled within each line by applying it to a sample-and hold device, so that each line is divided into a number of sampling periods (T) or pixels, yielding a time sampled analog signal. In the discussion that follows the terms sampling period, pixel, and pixel time are used interchangeably 
     In general a 2-D recursive (IIR) digital filter transfer function is given by: E1 ? ##STR1## where M×N is the order of the filter and {a ij  } and {b ij  } are the filter coefficients and are determined from a design procedure wherein stability and specifications are considered. (Usually b 00  =1 in the above equation.) 
     In the time domain, z 2   -1  corresponds to a time delay of one pixel time, while z 1   -1  corresponds to a row delay which in the case of a raster scanned image is a delay of one line scanning period. Note that whether or not an optional sample-and-hold device is used, the input signal may be thought of conceptually as being divided into a number of pixels. 
     Note that an FIR (finite impulse response) filter results when all of the b ij  =0 (except for b 00  =1), and an IIR filter results otherwise. A non-recursive (FIR) filter will always be stable. 
     Corresponding to equation (1) in the time domain is the so-called recursive equation which relates the output sample of the filter y(m,n) to the input sample x(m,n) occurring at the same point in time. ##EQU1## where again M×N is the order of the filter and {{a ij  }, {b ij  }} is the set of design coefficients. We have made the assumption of a casual system in equations (1) and (2) so that the output signal y(m,n) is written as a function of the input signal x(m,n) and past values of the input signal and output signal, x(m,n), y(m,n) respectively. 
     We can write equation (2) in the z-domain as ##EQU2## 
     Multiplying by z 2   -1  (thus introducing one pixel of latency in the output) and re-arranging yields ##EQU3## for i=1, . . . ,M. 
     The realization of the partial result z 2   -1  Y i  (z 1 , z 2 ) in terms of processing elements (PE&#39;s) 3, based on equation (6), is shown in FIG. 1. Shown below in FIG. 1 is the general block diagram symbol for a processing element with the input-output relationship Y PE  =(w 1  X IN  +W 2  Y IN  +Y&#39; IN )z 2   -1  indicated. 
     In FIG. 2, the internal structure of the general processing element is shown. Each PE consists of two coefficient multipliers 25, an adder 26, and a pixel delay 27, connected as shown. Note that w 1  and w 2  correspond to some specific filter design coefficients in the set {{a ij  },{b ij  }} that are required in a particular PE. 
     The overall 2-D semi-systolic filter structure based on equations (4), (5) and (6) is realized as shown in FIG. 3. There the following should be noted. Firstly, the output signal of the structure Y*(z 1 ,z 2 ) has been denoted with an asterisk because the output signal is produced with a latency of one pixel time and is actually z 2   -1  Y(z 1 ,z 2 ), which corresponds to y(m,n-l) in the time domain rather than y(m,n). To remove the effects of this latency on the feedback, the first line delay 2, in the feedback path must, in practice, be shortened by one pixel time and is thus denoted z 1   -1* . Also, because a latent (by one pixel time) signal is fed back to the lowermost bank of PE&#39;s, the &#34;b&#34; coefficients occur in the order 0, . . . ,b 0j , . . . ,-b 01  from left to right, in that bank of PE&#39;s (as shown in FIG. 3). In FIG. 3, z 1   -1  is used to represent a row delay 1, which in the case of a raster scanned image is a delay of one line scanning period (63.5 μs in the NTSC television standard presently in use in the United States). 
     The invention will now be further illustrated by discussing a preferred embodiment of a 2×2 IIR filter structure. Taking M=N=2 in FIG. 3, we obtain the structure of FIG. 4 which shows the overall signal processing structure in block diagram form as composed of signal delay lines 1, 2, and processing element blocks 3. The signal delay lines 1, impart a delay of one horizontal line scanning period. A horizontal line scanning period is often denoted as 1H and corresponds to 63.5 μs in the NTSC system. The signal delay line 2, differs from those denoted 1, in the figure, only in that the time delay imparted is one pixel time less than one horizontal line scanning period (i.e. 1H-T where T is the pixel time). Partial results z 2   -1  Y O  (z 1 ,z 2 ), z 2   -1  Y 1  (z 1 ,z 2 ) and z 2   -1  Y 2  (z 1 ,z 2 ) are added by an operational amplifier summing circuit to produce the output Y*(z 1 ,z 2 ). 
     The signal delay lines are typically charge transfer devices (CCD&#39;s) and are operative to delay signals in the baseband or video frequency range (e.g. 0 to the vicinity of 5 MHz). Such devices are commercially available, an example being the Fairchild-Weston CCD321, fabricated in the buried-channel NMOS process. FIG. 5 shows a CCD 1H delay line. The clock system oscillator 4 is typically a crystal controlled oscillator, and is available commercially from many sources. Alternatively, if the system is to operate in a color television receiver, the clock system oscillator signal may be generated by supplying the color subcarrier oscillator (typically 3.58 MHz) to a frequency multiplier circuit to obtain a 14.31818 Mhz clock signal as would be required for a CCD with 910 stages, to obtain a delay of one horizontal line scanning period (1H). The clock signal from oscillator 4, is supplied to clock driver circuitry 5. The type of clock driver circuits used will be a function of the type of charge coupled device being used 6. These circuits will typically be TTL or CMOS family integrated circuit devices arranged to provide a system of two-phase clock signals, φ 1  and φ 2 , which are oppositely phased, 50% duty cycle, relatively square waves. These clock signals are suitable for effecting transfer of charge between stages of a charge transfer device type of delay line. In the Fairchild-Weston CCD321, which has a charge injection port at its input and a sample-and-hold circuit in its output amplifier, the two-phase system of clocks φ 1  and φ 2  is applied to the device to effect charge injection at the input as well as interstage charge transport and clocking of the sample-and-hold circuit in the output amplifier. (A sample-and-hold device in the output stage of a CCD is not essential to the operation of the present invention but offers the advantage of reducing the clock frequency components in the signals). The output signal from the CCD is a line delayed version of the input signal, which may contain components of the sampling clock frequencies and its harmonic frequencies. These undesirable frequency components in the output can be suppressed by a suitable low pass filter circuit 7. A passive low pass filter suitable for this purpose based on an arrangement of inductors and capacitors is shown in FIG. 5B. A gain stage can be used in conjunction with such a passive low pass filter to compensate for losses. 
     A more complex system of clock driver (logic) circuits for charge-coupled devices which has the advantage of reducing noise is described in U.S. Pat. No. 4,158,181, Jun. 12, 1979, by Balopole. 
     The delay element has been described in a typical embodiment based on the use of a CCD, however, other conventional devices are well-known for being able to produce a delay of 1H. Such devices include those that are based on acoustic-wave propagation such as glass block devices. 
     All of the processing elements (PE&#39;s) 3, in the preferred embodiment of FIG. 4 are identical. A possible embodiment of a PE is shown in FIG. 6. The op-amp 8 is configured as an inverting summer. Resistors R a  and R b  are variable while resistors R&#39; and R f  are fixed such that R&#39;=R f . The output of the op amp will produce the sum (-R f  /R a )X IN  +(-R f  /R a )Y IN  +(-Y&#39; IN ) Comparing with the block diagram model of the PE shown in FIG. 2, w 1  =(-R f  /R a ) and w 2  =(-R f  /R b ). The op-amp 8, should be a wide-band type operational amplifier suitable for the video baseband signal range being processed. Power bandwidth and slew-rate and settling time specifications must also be adequate for proper operation. The settling time specification should be sufficiently small as to allow for rapid settling within the pixel time of the signal being processed. Typical devices for NTSC application are the LM318, and the EL2020C (Elantec). The EL2020C is a high-speed Monolithic (50 MHz) current feedback type op-amp, specifically designed for video and other applications that require high slew rates, fast settling times and low power consumption. Higher performance (200 MHz) current feedback op-amps specifically designed for HDTV and other comparable high-speed applications such as the EL400 (Elantec) are available in an 8 pin mini-DIP package that is pinout compatible with the EL2020C. 
     Note that w 1  and w 2  referred to above correspond to some specific filter design coefficients in the set {{a ij  },{b ij  }} that are required in a particular PE. In general these coefficients may be either positive or negative and the inversion introduced by the op-amp summing circuit must be taken into account. Signals X IN , Y IN , and Y&#39; IN  in FIG. 6 should be assigned the appropriate algebraic sign so as to correctly satisfy equations (4), (5), and (6) for the overall structure. Sign change of a signal may be accomplished, wherever required, by means of an inverter circuit. Inverter circuitry may be based on op-amps as shown in FIG. 7. A 4.7 pF capacitor 11, is required in the feedback loop to prevent oscillations in the output when a wideband amplifier such as the LM318 is used. 
     In the embodiment being described, a passive delay line 9, (in FIG. 6), is used to implement a pixel delay (denoted z 2   -1  in the z-domain). In the time domain, a signal x(m,n) is delayed by one pixel time resulting in a delayed version of the signal x(m,n-1). A typical passive delay line device is the type 1514-70D (Data Delay Devices). Lumped constant delay lines of this type are widely used in a number of electronic applications. For instance, in color TV receivers the luminance signal is delayed to synchronize it with the chrominance signal. The type 1514-70D delay line has a delay time T d  =70 ns and a characteristic impedance z 0  =250Ω. Accordingly the delay is terminated in a matching (in this case 250 Ω) resistor 10. This type of delay line is available in a four pin SIP package that does not take up much circuit board space. Lumped constant delay lines with a clock input and sample-and-hold output are available in eight pin DIP packages (e.g. Reticon 5100 series) and are also suited to the embodiment being described. 
     Photographs of a constructed prototype of a preferred embodiment of a 2×2 IIR filter apparatus such as has just been described are shown in FIGS. 8A and 8B. The filter circuitry without line delays is shown in FIG. 8B while FIG. 8A shows two circuit boards, each board containing two line delays of the filter. 
     There are alternative methods of realizing the processing element whose block diagram is shown in FIG. 2. We will now consider the processing element as embodied in FIG. 9, in which the coefficient values are digitally programmable and in which the pixel delay is implemented as a cascade of two sample-and-hold circuits. With reference to FIG. 9, it consists of two n-bit registers 12, for storing the filter coefficients, digital-to-analog converters 13, to convert the digital coefficient values w 1  and w 2 , into analog form and analog multipliers 14, which produce the products w 1  X IN  and w 2  Y IN . These two products together with Y&#39; IN  are summed by an inverting operational amplifier summer to obtain the direct sum (-w 1  X IN )+(-w 2  Y IN )+(-Y&#39; IN ). The signal represented by this sum is then delayed by one pixel time. In the circuitry shown in FIG. 9 this signal delay is accomplished as follows: The signal voltage is stored on capacitor C1 when sw1 closes during the high level of the clock waveform φ A . During the high level of φ B , sw2 closes and transfers the signal voltage that was stored on C1 to C2. During flyback, i.e. the transition form one line scan period to the next, the horizontal sync. pulse φ SYNC  is used to close the two analog switches sw3 and sw4 to discharge capacitors C1 and C2, if it is desired to set zero initial conditions at the output of the processing element at the beginning of each line scan period. 
     The invention, as described so far, requires line delays of different lengths. For example, in FIG. 4, there are line delays 1, which impart a delay of 1H (one line scan period) while the line delay 2, imparts a delay which is one pixel time less than 1H. The need for a non-standard length line delay may make implementation impractical. In FIG. 10 we illustrate how this situation can be remedied within the present invention. FIG. 10 is similar to FIG. 4 in that the case M=N=2 is depicted. However in FIG. 10 all line delays 1, are of standard length 1H, while the PE&#39;s are of two differing types which will be referred to as type I and type II. Type I PE&#39;s 3, are used everywhere except at the end of each bank of PE&#39;s, where a type II PE 15, is used instead. A type I PE has the structure given in FIG. 2 and may be embodied in practical hardware as suggested in FIG. 6. Thus type I PE&#39;s are the same as those that we have been describing until now. 
     A type II PE has the structure given in FIG. 11, from which it is seen that a type II PE is simply a type I PE without a pixel delay stage (often denoted z 2   -1 ). Accordingly a type II PE may be embodied in practical hardware in the same way as a type I PE, but with the components for producing a pixel delay being excluded. 
     Another change, shown in FIG. 10, which may be made for greater ease of construction with off-the-shelf components, is the introduction of dual broadcast lines, allowing either non-inverted or inverted data to be sent to each PE. (Note that lines through which PE&#39;s receive the same data in parallel are called broadcast lines.) In FIG. 10 each of these dual broadcast lines is depicted as having a non-inverted data line 16, and a data line 17 on which the data is inverted in sign. 
     The local variables of the PE&#39;s are the filter coefficients, that is, the set of numbers {{a ij  }, {b ij  }}, in equation (1) that are chosen to approximate the desired filter response. The use of dual broadcast lines provides a convenient method for assigning the correct sign to the data reaching each PE, assuming that local variables are set within each PE without regard to sign. 
     Many applications of the characteristic type of apparatus that result from the method of the present invention are possible. These applications include television raster scanned imaging systems in a diversity of areas, including those of broadcast television, industrial video, and biomedical video. FIG. 12 depicts a simple application in which 2-D signal processing apparatus of the type arising out of the present invention is inserted into the video circuitry of a black and white television receiver. With reference to FIG. 12, the detected video signal available at the emitter follower 18, at approximately a one volt peak to peak level, is supplied (typically AC coupled as shown) to the input of the 2-D signal processing apparatus 19, which implements a given filtering technique. The filtered signal is in turn sent to the final video output stage 20. 
     The foregoing descriptions of preferred embodiments of the present invention have been presented for the purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching. Moreover, it should not be construed that the present invention is limited to those applications which involve raster scanned images in the context of a particular standard, which is either existing or envisioned (i.e. NTSC, PAL, etc.).