IMAGE SIGNAL PROCESSOR

Abstract: An image signal processor which includes a local image register for receiving local image area data of m rows × n columns pixels, and a expansion use register of m row × l column pixels coupled to the output of the local image register. Thereby, expansion of local image area, and parallel processing can be readily conducted.

Description: This invention relates to an image signal processor, and more particularly to an architecture of an image signal processor in which local image processing such as spatial convolution, non-linear neighbor arithmetic operations can be conducted at high speed and also, expansion of local image area and parallel processing by use of multi-processor can be readily conducted. Generally speaking, image processing includes the following five steps which are sequentially conducted. That is, (i) observation, (ii) sampling/quantizing/coding, (iii) preprocessing, (iv) feature extraction, (v) object recognition. An object is observed by, generally, a video camera, whose image output is then digitized. In this case, the digitized image includes random noise due to camera characteristics and light reflection. Therefore, preprocessing is used to remove the unwanted noise component. After this processing, features are extracted from the preprocessed image signal and, thereafter, the extracted features are used to identify the object which was observed by a video camera. In such image processing, the preprocessing consumes the most time since it handles a huge amount of digital data which represents an image. Current Von Newman type computers are not good at such processing. Then, several trials have been conducted to realize high-speed image signal processing by parallel-processing image signal data, but it is extremely difficult to parallel-process whole data of a picture frame. Local parallel image processing, which handles local image data of m-by-n picture elements (pixels) of a picture frame, is applicable to wider processing such as averaging, differential operation, data transformation and so on, and the size of circuitry therefor is relatively small. Therefore, development of LSI for use in such local parallel image processing is actively being conducted. A conventional local parallel processing type image signal processor has a specific structure which is exclusively used for each image processing function and therefore, in general, does not have general purpose-use-structure and expansion function. Generally speaking, a local image signal processor is one which picks up local image area data of a certain proper size out of input image data, and which applies a calculation to such local image area data. That is, a local image signal processor handles image processing of a whole picture frame by scanning a window which simply covers a local image area over the whole area of a picture frame. Among image signal processing methods, there are many types of processing which are conducted by local processing such as averaging, differential operation, feature extraction and so on. These types of processing have different complexities from one another according to configuration and size of the local image area. Generally, such image processing is conducted on a local area of approximately 3-by-3 through 16-by-16 pixels. One example of a conventional local image signal processor is disclosed in "Image signal processor computes fast enough for gray-scale video", Tadashi Fukushima, Electronic Design, October 4, 1984, pages 209 through 215. The image signal processor disclosed in Electronic Design has a specific exclusive use structure, but does not have general purpose use structure, and further, requires many peripheral/additional circuits to realize expansion processing. The paper "Comparison of parallel architectures for image processing circuits" by Ruetz et al (IEEE International Symposium on Circuits and Systems, San Jose, 5-7 May 1986, Vol. 2, pages 732-7) compares such special-purpose processors with programmable general purpose processors for image processing purposes and its conclusions favour special-purpose processors for their lower hardware complexity. A general purpose image processor is described in EP-A-0189943 (Hitachi, Ltd). This image processor may be provided as an LSI circuit and is capable of changing (notably expanding) the size of the local area being processed. For expanded local area processing a number of local area processors are used together and adjoining sub-areas are proposed simultaneously by the respective processors. The results of the processing steps are then passed serially from one processor to the next with each processor adding its result to the value received from the preceding processor(s). The present invention, therefore, has its principal object the provision of an improved image signal processor which particularly realizes expansion of local image area with high speed operation and has a simple architecture suitable for LSI. This object of the invention is accomplished by an image processor according to the present invention, which processor includes: m image data input ports for receiving local area image data; a local area image register consisting of m rows of sequential memories, each row of sequential memories receiving data from a respective image data input port and each row of sequential memories being adapted to hold image data relating to n pixels; and a calculation unit coupled to the local area image register for performing calculations based upon local area image data therefrom, the calculation unit receiving (m x n) local area image data from the local area image register and having an input terminal for receiving external data; characterised in that there is further provided: an expansion use register comprising m rows of memories, each expansion use register row receiving data from the output of a respective row of sequential memories of the local area image register, for delaying the output of said row of sequential memories, and each expansion use register row being adapted to hold image data relating to one pixel; and m image data output ports for outputting local area image data delayed by passage through the local area image register and expansion use register. In a specific embodiment, the local image register comprises (m x n) one-pixel shift registers m may be 3 and n may be 3. The expansion use register comprises m one-bit shift registers. In this case, m may be 3. According to the present invention as described herein, the following benefits, among others, are obtained. (1) It is possible to obtain an image signal processor which can realize high speed local image processing. (2) It is possible to obtain an image signal processor which can readily realize expansion processing of local image area using an extremely simple structure. (3) It is possible to obtain an image signal processor which has general purpose use structure by program control. The features and advantages of the present invention will be more clearly understood from the following description of embodiments thereof, given by way of example, taken in conjunction with the accompany drawings, in which: Fig. 1 shows a principle of local image processing on which the present invention is based; Fig. 2 is a block diagram of one embodiment of a local image processor according to the invention; Fig. 3 is a block diagram of a local image processing system using a processor according to said one embodiment, which has a local image area expansion function; Fig. 4 illustrates the operation of the local image processor shown in Fig. 3; Fig. 5 is a detailed block diagram of a local image processor of one embodiment; Fig. 6 is a block diagram of a local image processing system making a further application of the one embodiment processor; Fig. 7 is a timing chart for explaining the operation of the local image processor shown in Fig. 6; Fig. 8 is a block diagram of a local image processing system making a still further application of the one embodiment processor; Fig. 9 illustrates the operation of the local image processor shown in Fig. 8; and Fig. 10 is a timing chart for explaining the operation of the local image processor shown in Fig. 8. Detailed Description of the Invention The invention is explained with reference to the drawings. Fig. 1 illustrate a principle of local image processing. Local image processing method is mostly applied to digital image processing systems such as edge detection, smoothing, etc. The data of each pixel and its neighboring pixels in the local window 10 are gathered and processed by processor 12 so that output pixel 14 of output image 16 is obtained, as shown in Fig. 1. The local window 10 scans sequentially over the frame picture of input image 18 one after the other, controlled by scanning controller 20. For realtime image processing, a very high-speed processor is indispensable. In the case of the NTSC system with 512 × 512 pixels in one frame, the sampling rate of image signals is about 90 nS per pixel. In this case the processor must complete the calculation of all the data of a local window within 90 nS. The local window generally covers 3 × 3 through 16 × 16 picture elements. The larger local window we choose, the higher processing speed we need. The present invention is a simple but powerful realtime image signal processor. Because of the development of the novel local image register and the pipeline register, the image signal processor of this invention can be operated in multi-chip mode in such image processing applications that require much higher speed of the enlarged local window. The speed can easily be increased to twice or more when two or more image signal processors are used, and simultaneously the local window can be enlarged without any time loss. Fig. 2 shows one embodiment of a local image signal processor for handling a local image area of size 3-by-3 pixels or 3 rows × 3 columns according to the invention. Input image signals 18 are applied, directly and indirectly through one line shift registers 22, 24, to local image signal processor 12 which has input terminals 26, 28, 30, output terminals 32, 34, 36, external data input terminal 38, and calculation result output terminal 40. One pixel shift registers 42 ∼ 58 receive image data from terminals 26 ∼ 36 and store local image data of (m) rows × (n) columns or (m) × (n) pixels. One pixel image data comprises 8 bits. Another set of one pixel shift registers 60 ∼ 64 receives the output from series-connected registers 42 ∼ 46, 48 ∼ 52, 54 ∼ 58, respectively to produce outputs thereof to terminals 32 ∼ 36. Outputs of shift registers 42 ∼ 58 and external data from terminal 38 are applied to calculation unit 66 for calculation. Output of calculation unit 66 is applied to terminal 40. Image data 18 are sequentially read out one pixel by one pixel by the scanning of one input image, and are applied to shift register 42. Shift register 48 receives data which is one-line-delayed by one-line shift register 22 compared with the data which is applied to shift register 42. Shift register 54 receives data which is two-line-delayed by one-line shift registers 22, 24 compared with the data which is applied to shift register 42. As stated above, three kinds of data, which are delayed by one line, respectively, are applied to shift registers 42, 48, 54 and then, transferred to shift registers 44, 50, 56 and 46, 52, 58 respectively so that image data are transferred one pixel by one pixel. By such an operation, image data from the input image are re-constructed by shift registers 42 ∼ 58 as local image area data of 3-by-3 pixels or 3 rows × 3 columns. These shift registers 42 ∼ 58 are referred to as local image register 59. Local area data are processed by calculation unit 66 so that image processing of the whole area is possible. Shift registers 60, 62, 64 are referred to as expansion use register 65. There is a relevant U.S. patent application, Serial No. 682,321, filed December 17, 1984 (now US patent 4635292) which includes a similar circuit structure to the Fig. 2 structure, but does not include expansion use register 65. Fig. 3 shows a case in which image processing of an expanded local are is conducted by use of a plurality of local image signal processors 12. In this Fig. 3 embodiment, expanded local image processing of 6-by-6 pixels or 6 rows × 6 columns is possible by use of four processors 12A ∼ 12D of 3 rows × 3 columns. The structure of each processor is the same as shown in Fig. 2. Input image signal 18 is applied through image signal input terminal 68 to image signal processor 12A. The structure of processor 12A and its peripherals is the same as shown in Fig. 2. Output terminals 32, 34, 36, 40 of processor 12A are connected to input terminals 26, 28, 30, 38 of processor 12B. The output terminal 40 of processor 12B is connected to the external signal input terminal 38 of processor 12C. The output of one-line shift register 24 is applied through two pixels shift register 70, one-line shift register 72, to input terminal 26 of processor 12C. The output of one-line shift register 72 is applied through one-line shift register 74 to input terminal 28 of processor 12C. The output of one-line shift register 74 is applied through one-line shift register 76 to input terminal 30 of processor 12C. The output terminals 32, 34, 36, 40 of processor 12C are connected to input terminals 26, 28, 30, 38 of processor 12D, respectively. Final output 78 is produced from output terminal 40 of processor 12D. In Fig. 4, an image to be processed is designated by numeral 80. Local areas 82D ∼ 82A are supplied to and stored in local image signal processors 12A ∼ 12D simultaneously at an arbitrary timing. Local areas 82A, 82B have a space of one pixel therebetween in the horizontal direction. Local areas 82C, 82D have a space of one pixel therebetween in the horizontal direction. As shown in Fig. 3, calculation result output signals of processors 12A ∼ 12C are applied through output terminals 40 to external data input terminals 38 of the next stage processors 12B ∼ 12D. Therefore, in the case of spatial convolution calculation etc., the convolution calculation result of a certain stage processor is ANDed, at the next timing, with the convolution calculation result of the next stage processor. By transferring the calculation result of each processor to the next stage processor in the scanning direction, the output 78 of the final stage processor 12D becomes the calculation result of an expanded local area (in this case, 6 rows × 6 columns). The number of bit for shift register 70 differs according to the number of local image signal processor to be used. In the Fig. 3 structure, four local image processors 12A ∼ 12D do not produce their outputs simultaneously. That is, outputs from terminals 32, 34, 36 of processor 12A are delayed by expansion use shift resistors 60, 62, 64 (see Fig. 2) by one pixel (one bit). The output from terminal 40 of processor 12A is also delayed by the calculation operation of calculation unit 66. Therefore, the output from terminal 40 of processor 12B is delayed compared with the output from terminal 40 of processor 12A. Similarly, the output from terminal 40 of processor 12C is delayed compared to the output from terminal 40 of processor 12B, and the output from terminal 40 of processor 12D is delayed compared to the output from terminal 40 of processor 12C. As stated above, the four outputs of processor 12A ∼ 12D are outputted at different timing. Therefore, for example, the output of processor 12A is ANDed with the outputs from local image register 59 (see Fig. 2) of processor 12B in calculation unit 66 of processor 12B. This means that there is no need to provide a peripheral circuit for local image processing. As explained above, expansion processing of local area can be realized without external additional circuits by use of a plurality of image signal processors, each of which includes local area registers of (m) rows × (n) columns and expansion use register of (m) rows × (n) columns. Fig. 5 shows an example of an architecture of an image signal processor according to the present invention. Local image register 84, to which image data 18 is applied in parallel manner, stores local image area data of (m) rows × (n) columns. Expansion use image register 86 receives the output signal of local image register 84, and shifts the image data in order, and produces image data output 88. The local image register 84 and expansion use image register 86 are driven by image read-in clock 90 from clock control circuit 92. The shift operation of expansion use image register 86 is controlled by expansion control signal 94 from input terminal 96. When the expansion control signal is supplied, the expansion use image register 86 is set in shift mode wherein expansion processing of a local image area is conducted in a pipeline manner. When the expansion control signal is not supplied, the expansion use image register 86 is set in pass mode or through mode wherein shift operation is not conducted. Arithmetic unit (adder and subtracter) 98 and multiplier 100 receive signals selected by selector 102, 104, 106 and conduct their calculation operations. Data register 108, 110 stores the calculation result of arithmetic unit 98. Data register 112 stores input data 114 from the data input terminal. Data register 116 stores the calculation result of multiplier 100. Data register 118 stores output data from the data register 110. Output control circuit 120 allows data signal output 122 to pass therethough only during a specified duration according to control signal 124 from clock control circuit 92. Output clock signal 126, which is generated from clock control circuit 92, is used for having data signal output 122 read in external register (not shown). Program memory 128 stores image processing program. When image processing is conducted, the program is read out by program control circuit 130 and then, each block is controlled by such read-out program. Each block is operated by clock signals from clock control circuit 92. The clock control circuit 92 receives system clock signal 132, program start signal 134, parallel control signal 136 and produces the above-stated output clock signal 126 and control clock signal for each block. It is possible to write in program memory 128 an address signal for reading out data of an arbitrary one pixel in local image register 84 which stores local area data of m rows × n columns, calculation control signal of arithmetic unit (adder and subtracter) 98, control signals for selectors 102, 104, 106, write-in control signals for registers 112, 110, 116, multiplifying numeral of multiplier 100 and so on. If these are combined to thereby prepare an image processing program, arbitrary calculations on local image area data which are read in local image register 84 can be carried out at high speed. As explained above, an image processing program, which is written in program memory 128, is executed on one local image area data. The calculation result which is stored in register 110 is transferred to output register 118 and then, outputted through output control circuit 120 as data output signal 122. Then, new local image area data is read in local image register 84. By repeating such operations in order, local parallel image processing of a whole image is conducted. In the case that the Fig. 5 architecture is made as LSI, the size of local image register 84 is limited to approximately 3 × 3 ∼ 5 × 5 from the view point of integration density. On the other hand, in the case of local parallel processing of an image, generally, a local image register of approximately 3 × 3 ∼ 16 × 16 pixels is used. If an image signal processor having a local image register of 3 × 3 pixels used for local parallel processing which handles local image area of 12 × 12 pixels, complex external circuitry is required other than 16 image signal processors. In this invention, such external circuitry is not necessitated to conduct the above-stated processing. Further, an image signal processor according to the invention has a structure which readily enables parallel processing by use of multi-processors. In the case that local parallel processing is conducted during a given period and, processing speed is beyond the performance of an image signal processor, parallel processing must be conducted by using a plurality of processors. In this case, generally, complex external circuitry is required, but, in this invention, parallel processing can be realized without external circuitry. Fig. 6 shows an example wherein parallel processing is conducted by using two image signal processors of the present invention. Image data from input terminal 68 is applied directly and indirectly through one-line register 22 and one-line registers 22, 24 to image signal processor 12X as input data 18a ∼ 18c. Program start signal 134 from input terminal 138 and parallel control signal 136 from input terminal 140 are applied to image signal processor 12X. Image data 18a ∼ 18c and program start signal 134 are also applied to image signal processor 12Y. Parallel control signal 136 from input terminal 142 is also applied to image signal processor 12Y. The data outputs 122 of image signal processors 12X, 12Y are applied to OR gates 144, 146, respectively. These OR gates 144, 146 also receive output clocks 126 of image signal processors 12X, 12Y, respectively. The outputs 148, 150 of OR gates 144, 146 are applied to external register 152 which then produces output 154. Each of the image signal processors 12X, 12Y includes a local image register 84 of 3 × 3 pixels (see Fig. 5). The image data input terminals of image signal processors 12X, 12Y receive image data of three lines simultaneously by use of one line registers 22, 24. Although the same image data is applied to image signal processors 12X, 12Y, local image area data is read in each local image register 84 of each image signal processor 12X (12Y), one by one. Fig. 7 shows voltage wave forms of the principal portions of the Fig. 6 structure. In Fig. 7 (a) is shown a program start signal 134. The processing operation of image signal processors 12X, 12Y is initiated and read-in of image input data 18 [see, (b) of Fig. 7] to local image register 84 is initiated in synchronism with the program start signal 134. (c), (d) show parallel control signals 136 to be applied to image signal processors 12X, 12Y. These two parallel control signals 136 are opposite in phase to each other. These parallel control signals 136 are applied to clock control circuits 92 of image signal processors 12X, 12Y, respectively and control read-in clock of local image register 84, clocks for program start timing, output control circuit, external register. That is, image data read-in to local image register 84 is conducted one by one as shown in (e), (f). According to this, the program of each processor 12X (12Y) is initiated one by one and executed so that calculation is carried out to read-in local image area data. The calculation result of each image signal processor is outputted one by one in synchronism with program completion. The data output is outputted for a given period by output control circuit 120 so that (g), (i) wave forms are obtained. Data output signals 122 are added by OR gate 144 (h), (i) show clock signals 126 which are outputted from the image signal processors and used for read-in to external register 152. The clock signals 126 are also added by OR gate 146. Added calculation result 148 and added clock signal 150 are applied to external register 152 to thereby produce processed data 154 which is consecutive. Fig. 7 (k) shows a signal 154 (see Fig. 6) which is inputted to the external register 152. In the case that the image signal processor is formed by an ECL gate, an OR gate can be formed by wired OR which is constructed by simply wiring so that OR gates 144, 146 are not necessitated. As explained above, parallel processing is possible using a simple structure. Fig. 8 shows a local image processing system being a still further application of the invention. In some applications, there is a need to handle the data of an enlarged local window; for example, an array of 12 × 12 pixels as shown in Fig. 9. The processor can be used in this case, because of its novel pipeline architecture. For instance, the spatial filter with the expanded local window of 12 × 12 pixels is This convoluted equation is re-written down with sixteen partial convolutions as The enlarged local window is further divided into sub-windows-a, b, c, .... p. Each sub-window has a data size of 3 × 3, the same as the local-image register of the processor. Sixteen units A, B, C, ...., P process the sub-window data, one by one. As shown in Fig. 8, sixteen units are connected serially. The unit A handles the image data of the sub-window-a, and calculates the partial convolution; G₁. The partial convolution; G₁ is taken out at the next clock cycle of an input timing of the image data. The image data through the local-image register are internally delayed by one clock cycle by the pipeline register, and are transferred to the next unit. Then the image data of sub-windows are taken in to each unit with shifting by one clock cycle each other. So the data of the partial convolution; G₁ and the image data of the sub-window-b are supplied to the unit B at the same timing. Then the unit B can simultaneously achieve the convolution of G₂ and the summation of G₁ and G₂. With the same method, the unit C achieves the convolution of G₃ and the summation of (G₁+G₂) and G₃. The unit P achieves the convolution of G₁₆ and the summation of (G₁+ G₂+ .... + G₁₅) and G₁₆. This data output from the unit P is just G (X, Y). This pipelining technology makes it possible to expand the local window with no time loss nor the use of any other processing unit. Pipelining and the parallelism of the processor can be utilized simultaneously.

Claims: An image signal processor comprising: m image data input ports (26,28,30) for receiving local area image data; a local area image register consisting of m rows of sequential memories (42-46,48-52,54-58), each row of sequential memories receiving data from a respective image data input port (26,28,30) and each row of sequential memories being adapted to hold image data relating to n pixels; and a calculation unit (66) coupled to the local area image register for performing calculations based upon local area image data therefrom, the calculation unit (66) receiving (m x n) local area image data from the local area image register and having an input terminal (38) for receiving external data; characterised in that there is further provided: an expansion use register comprising m rows of memories, each expansion use register row receiving data from the output of a respective row of sequential memories of the local area image register, for delaying the output of said row of sequential memories, and each expansion use register row being adapted to hold image data relating to one pixel; and m image data output ports for outputting local area image data delayed by passage through the local area image register and expansion use register. The image signal processor of claim 1, wherein said local area image register comprises (m x n) one-pixel shift registers. The image signal processor of claim 2, wherein m is 3 and n is 3. The image signal processor of claim 1, wherein said expansion use register comprises m one-pixel shift registers. The image signal processor of claim 4, wherein m is 3. The image signal processor according to any previous claim, wherein the calculation unit (66) comprises an arithmetic unit (98) and the processor further comprises: first control means (92) for supplying clock signals (90) to the local area image register (84) and expansion use register (86) in accordance with a parallel control signal (136); a register (118) coupled to the arithmetic unit (66) for storing the output of said arithmetic unit (98); and second control means (120) for passing the output of said register (118) therethrough for a predetermined period, said period being determined by said first control means (92) according to said parallel control signal (136). An image signal processing system, comprising: at least a first and a second image signal processor according to any one of claims 1 to 6, the first signal processor being adapted to receive image data which is applied to the local area image register of said first image signal processor and the second image signal processor being adapted to receive outputs from the expansion use register (32,34,36) and calculation unit (40) of said first image signal processor, the output (32,34,36) of the expansion use register of the first image signal processor being applied to the local area image register of the second image signal processor and the output of the calculation unit of the first image signal processor being applied to the calculation unit of said second image signal processor, whereby the output of the calculation unit of the first image signal processor and the output of the local area image register of the second image signal processor are applied to and used in calculations by the calculation unit of the second image signal processor.

Inventors: AONO, KUNITOSHI, MARUYAMA, MASAKATSU, MORI, TOSHIKI, YAMADA, HARUYASU
Applicants: MATSUSHITA ELECTRIC INDUSTRIAL CO., LTD.
Application year: 1986
Main classification: G06F15
IPC codes: G06T5