APPARATUS AND METHOD FOR THE COMPRESSION OF IMAGE DATA

Abstract: Apparatus and method for the compression of three-dimensional image data derived from different perspectives of an object field. The data is arranged in a perspective array having at least two dimensions with respective axes, a first dimension representing distance across the object field and a second dimension representing perspective of the object field. Points in the object field provide corresponding loci in the perspective array. Each locus and its skew with respect to the axis of the perspective dimension is found by correlating the associated image data. The skew provides a measure of the range or depth of the related point in the object field. The invention has applications in machine vision for robots and collision avoidance. It may also be applied to data compression for transmission of three-dimensional images and to the manipulation of such images.

Description: Apparatus and Method for the Compression of Image Data This invention relates to the encoding of three-dimensional images and specifically to the compression of such image data into a format useful for transmission, storage and/or analysis. The use of a lenticular screen comprising side-by-side elon¬ gate cylindrical lens elements or alternatively, an array of substantially rotationally symmetric raicro-lenses arranged on a two-dimensional grid, for the recording, replication and/or viewing of an auto-stereoscopic image is well known. U.S.Pat.No.4 , 552 ,442 (Street) provides an imaging system for the recording of such images, whilst U.S.Pat.No.4,668 , 063 ( Street) provides a modified form of such an image suitable for the reproduction of viewable images. In patent application PCT/GB91/01920 (Street), corresponding to ϋ.K.Pat.Appl .No.9024043.3, a method for processing cellular images of the kind typically associated with an array of small lens elements is described and the concept of a multi-valued picture element (or super-pixel) is developed. Such a super- pixel has not only three components (conventionally RGB values), but each of these is in fact a function of angle of view of the original scene. Such a definition of a pixel vec¬ tor function allows for the full encoding of a three- dimensional image over the allowable range of the "angle" selection variable. An entire image comprised of a large num¬ ber of such super-pixels can be processed in a variety of ways before being used to reproduce a viewable 3-D image. In its basic form the volume of data comprising such a 3-D pixellated image is many times that needed for the equivalent two- dimensional image, which would comprise only one perspective thereof. For an auto-stereoscopic image displaying only horizontal parallax, a factor of 50 would be quite practical, whereas a full parallax image can increase this to several thousand. It is a principal objective of this invention to reduce the volume of data needed to encode a typical 3-D image, to do this in a manner which does not substantially degrade the image when reconstructed, and to provide a means through which the range of subject matter within the image is easily extracted. In the prior art, it is common practice to establish the range of the key subject within an object field (or scene) by correlating the image regions in two different views which correspond to this subject matter. Typically these two views or images are obtained from two viewpoints spaced some known distance apart. By displacing the pattern of one such image with respect to the other within an image processing circuit or system designed to compare two patterns, an optimal match may be obtained. The displacement required to obtain this match corresponds directly and simply to the average distance of the key subject from the range finding apparatus. A good example of such a range finding system is provided in U.S.Pat.No. ,185,191 (Stauffer), which describes a "through- the-lens" ranging system as might be used in photography or cinematography. The images to be compared are obtained in electronic form by using two interleaved arrays of detectors, such as a CCD array (charge coupled device), placed behind a corresponding array of small lens elements (microlenses) . In this arrangement each microlens is associated with one element of each array, in such a way that one array of detectors receives an image of the subject matter formed through one half of the camera lens, whilst the other array's image is formed by the other half of this lens. There is sufficient effective distance between these two image forming regions of the lens to provide a displacement of the corresponding image regions suitable for a range to be determined. Systems of the type described have two main limitations. Firstly, correlation is achieved for an image region as a whole. Where the image corresponds to subject matter which is essentially at one range and an average distance thereto is required, such as in an automatic focusing system for a camera, this is not a problem. However, if precise but different depth values are needed for a number of points in the same image field, for instance, where a robot must establish the shape of an object in three dimensions, serious difficulties will arise. The second limitation of conventional image correlation, which utilises a stereo pair of images, is that ambiguities may occur which result from the image content. For instance, a fence having regularly spaced slats could easily cause a false match, as each slat could match either of its near neighbours quite well within the image region of interest. It will be clear to those skilled in the art that, the more image data contained within the comparison process, the less likely an ambiguity, but the greater the computational task. This invention addresses the difficulties of the prior art methods by establishing the distance to every point within an object field as an individual entity. Range is not obtained by correlating one image pattern and a second, corresponding one. Rather, a substantial number of viewpoints is provided to each visible point of the object field within the region of interest, and correlation is obtained, between these corresponding views, for each point separately. By adopting this approach, the distance to any visible point on an object is defined right up to its edge, as is the distance to any visible background matter behind it. The main advantages of the current invention over the existing prior art are:- a) The efficient storage of 3-D image data. b) The ability to transmit a 3-D image with a minimal increase in bandwidth when compared to its 2-D equivalent. c) The extraction of identifiable depth co-ordinates from 3-D image data without complex correlation algorithms. d) The simple location and/or surface profiling of subject matter in three dimensions for applications such as robot vision. Thus according to the present invention there is provided apparatus for the compression of image information comprising three-dimensional image data input means for providing data derived from different perspectives of an object field; means for holding the data in a perspective array having at least two dimensions with respective axes, a first dimension representing distance across the object field and a second dimension representing perspective of the object field, each of a plurality of points in the object field providing a locus in the perspective array, said locus being skewed with respect to the axis of the perspective dimension to an extent dependent upon the depth of the point in the object field; and analyser means for analysing the data to produce respective depth values for the object points from the respective degrees of skew in the perspective array of the loci associated with those points. In one embodiment of the invention the image data input means comprises perspective separation means. The perspective separation means may comprise means for re¬ arranging three-dimensional image data to provide the perspective array. For example, the data may be generated by scanning a three- dimensional image recorded on film, as described in Patent Application PCT/GB91/01920. Alternatively, such image information may be recorded on video tape. It is also possible to synthesise, in whole or in part, three-dimensional image data using a computer, as opposed to using a real image. Where a real object field is provided, the perspective separation means preferably comprises image forming means; primary aperture means having substantial extent in at least one direction of perspective change; and image capture means. The perspective separation means may comprise an array of small convergent elements in close proximity to the image capture means, so that in use, image forming rays from different perspectives of a point in the object field impinge at respectively different positions on the image capture means. Alternatively, the perspective separation means may include a secondary aperture selection means intermediate the object field and image capture means selecting, in use, a portion of the image forming rays from the primary aperture means, whereby image forming rays from a substantial range of perspectives of the object field are captured sequentially. The secondary aperture selection means may comprise an aperture substantially smaller than the primary aperture means, said smaller aperture being movable relative to said primary aperture. Alternatively the secondary aperture selection means may comprise a shutter array having elements which are selectively operable. Preferably the image capture means comprises a CCD array. Advantageously the image capture means comprises spectral separation means. Preferably the analyser means includes means for identifying correspondence between adjacent image data elements in the perspective array. Advantageously the data represents a plurality of spectral components and said correspondence comprises spectral correspondence. Whilst the visible spectrum can be split into a number of colour bands or spectral components, there are applications in which other spectral components, for example in the infrared, can be very important. This can apply to robotics, seeing in bad weather and, particularly, to night vision. Where good correspondence is achieved, the skew of the locus in the perspective array of corresponding image data elements may be identified by track following the locus. In a preferred embodiment of the invention the analyser means comprises means for extracting data groups according to their respective degree of skew in the perspective array; correlation means for performing a correlation function on the data groups, the correlation means giving a likelihood function corresponding to the probability that the members of a data group represent the same object point; and means for deriving depth values for object points from the degree of skew associated with a maximum in the likelihood function. Advantageously the means for extracting data groups comprises means for shifting data in a plurality of shift registers, said data comprising all or part of the perspective array and representing a corresponding plurality of perspectives of the object field; means for shifting the data in all registers substantially in synchronism; and skew selection means. The skew selection means may comprise means for shifting data in respective registers by respectively different amounts, whereby a degree of skew in said data is selected prior to shifting said data in synchronism; and means for transferring successive data groups having the selected skew to the correlation means from a location fixed in each of a plurality of said shift registers. Alternatively the skew selection means may comprise means for extracting, substantially simultaneously, a plurality of data groups having a corresponding plurality of degrees of skew from a group of data elements extending in at least two dimensions of the perspective array. Advantageously the correlation means comprises means for converting data groups into analog signals; means for processing said signals to provide a signal corresponding to the likelihood function; and means for converting said function into suitable form for subsequent processing. The likelihood function may substantially comprise the variance of the data within each data group. Preferably the means for holding data in the perspective array comprises space for the data associated with at least six perspectives of the object field. Advantageously said space is sufficient for twelve perspectives of the object field. According to a further aspect of the invention a method for the compression of image information comprises providing three-dimensional image data derived from different perspectives of an object field; holding the data in a perspective array having at least two dimensions with respective axes, a first dimension representing distance across the object field and a second dimension representing perspective of the object field, each of a plurality of points in the object field providing a locus in the perspective array, said locus being skewed with respect to the axis of the perspective dimension to an extent dependent upon the depth of the point in the object field; and analysing the data to produce respective depth values for the object points from the respective degrees of skew in the perspective array of the loci associated with those points. Preferably analysing said data comprises producing an array of respective compressed picture elements, each element comprising at least one depth value. Advantageously analysing said data comprises locating discontinuities in depth value between neighbouring elements within the array; and assigning each discontinuity to the boundary of an object within the object field. The invention will now be described by way of example with reference to the accompanying drawings, in which:- Figures 1, 2A, 2B and 2C illustrate different three- dimensional imaging systems constructed in accordance with the invention. Figure 3 is a schematic diagram of a system for the correlation of image data in accordance with the invention. Figures 4A and 4B show in diagrammatic form how image data is arranged in a perspective array and related to the depth co¬ ordinates of points in accordance with the invention. Figure 5 is a schematic diagram of a circuit forming part of the system of Figure 3. In Figure 1, an object field 1 is shown diagrammatically to include two identifiable points A and B. A wide lens 2 with a matching wide primary aperture 3 forms respective image points A' # and B' at positions respectively behind and in front of a CCD array 4. A secondary movable aperture 5 shown as a small gap in a blocking screen 6, allows the image forming rays 7 and 8 of a single perspective of the object field 1 to be captured by array 4 independently of those from the other perspectives comprising the three-dimensional image. CCD support circuit 9 relays the image data associated with one perspective to data storage and processing circuit 10, before enabling a movement, relative to primary aperture 3, of aperture 5 through actuator 11 and receiving image data corresponding to the next perspective from the object field and thus achieving perspective separation sequentially. Figures 2A, 2B and 2C illustrate different aspects of embodiments, alternative to that of Figure 1, of means for providing three-dimensional image data input from three- dimensional images. Instead of a movable secondary aperture to achieve perspective separation, a lenticular screen, having an array of small convergent elements or lenses, is employed. In Figure 2A, a curved mirror 12 in combination with a semitransparent plane mirror 13, forms an image of object field 14 in the vicinity of lenticular screen 15 which is in close proximity to CCD array 16. Though not shown to scale for reasons of clarity, the width of individual lens elements comprising array 15 is selected to correspond to the spatial resolution required of the image to be captured. Accordingly, the number of elements within the CCD array 16 must be substantially higher. If there are 500 lens elements of the lenticular screen within the length of the CCD array and this has 6000 elements, then the system as illustrated can handle 12 perspective views of the object field simultaneously. Though not explicitly illustrated, the principle of perspective separation can clearly be accomplished by a hybrid solution containing both a secondary aperture selection means similar to that of Figure 1 and the lenslet array principles of the lenticular screens of Figures 2A to 2C. In this approach a group of perspectives is selected by the secondary aperture selection means. This may comprise a single aperture transmitting rays from several adjacent perspectives or a series of spaced apertures each transmitting rays from a single perspective. The precise choice will be application dependent and will determine the geometry of the lenticular screen in relation to the primary aperture. Although a moving selection aperture is illustrated in Figure 1, it is often advantageous to employ a solid state light switch such as a liquid crystal shutter array, the elements of which are selectively operable, thus sequencing different perspectives. In Figure 2B an imaging system comprising a retro-reflective screen in combination with a lenticular screen is shown. This avoids the effect of depth reversal and is fully disclosed in U.S.Pat. o.4,552,442 (Street). Point A in the object field which lies behind point B is first imaged to point A' by lens 17 which now lies in front of image point B' . Retro-reflective screen 18 redirects the image forming rays which are reflected by semitransparent mirror 19 and after passing through an optional scaling lens 20, two final three-dimensional image points A" and B" are formed having the correct spatial relationship. In Figure 2C a small portion of the lenticular screen is shown on a much enlarged scale, in order to illustrate the perspective separation function thereof. A pencil of rays 21 arrives at the surface of lenticular element 22 at a particular angle (typically related to a particular perspective) . A second pencil 23 arrives at the same element 22 but at a different angle. The two form quite separate images at the focus of element 22 which is in close proximity to the photosensitive elements of the image capture means, CCD array 24. It is quite appropriate, within the context of the embodiments of Figures 2A and 2B, to treat the group of CCD elements between points C and D and comprised of those associated directly with the element 22 as providing the image data for a multi-valued picture element (or super-pixel). Each CCD element within the group provides the value of the super- pixel for a specific but different perspective. The super- pixel represents, in fact, a function of a variable corresponding to perspective. It is assumed that for colour images a group of three or more CCD elements is used to provide this data for each perspective or that the same element may be used successively with different colour filters. In this case, liquid crystal technology may be used in combination with colour selective polarising filters to provide solid state colour selection. Such a system may clearly be combined with perspective selection. As in the embodiment of Figure 1, the data captured by the CCD array of Figure 2B is passed to image data storage and processing (compression) means 25. In an alternative embodiment of this invention, the means for receiving three- dimensional image data need not include an optical process, as this may have been performed previously, as taught by patent application PCT/GB91/01920 (Street). In this case, means for receiving such data can be confined to a suitable electronic interface. The format in which this data is arranged, and the processing to which it is subjected, is illustrated with the help of Figures 3, 4A, 4B and 5. In Figure 3 an image data analysis and compression system is shown in block schematic form. As each scan line of data from the full 3-D image is provided by the CCD and its associated support circuitry, or from another source such as an intermediate data storage facility, the digitised data representing each successive super-pixel is sent to the shift registers 26 through interface lines 27 and associated data switches 28. Each super-pixel comprises a range of different values and the data comprises a perspective array. All pixel values relating to one perspective are sent to a particular shift register. Thus, shift register 26A holds all the super- pixel values for the left-most perspective and register 26Z the corresponding values for the right-most perspective. For the sake of clarity, line 27A is shown as a single connection and, as such, can transfer a binary slice image (one having the value of zero or unity for any particular perspective of each super-pixel). In practice, these lines will be high speed serial or comprise many parallel connections. The switch 28A, register 26A and all other data storage and manipulation modules in Figure 3 are shown as having depth. This simply represents the fact that each super- pixel value may comprise a number of data bits in practice. Typically 24 bits is used to provide a high quality rendition of a "real world" image. Therefor, in this case, to transfer data having more than one bit of depth, the lines 27, and all other data transfer lines, must themselves each represent the appropriate number of physical connections or a serial line, the data from which is converted into parallel form at the corresponding data inputs, such as those of switches 28. Timing and control module 29 controls switches 28 through line 30. When the data associated with one scan line has been transferred to registers 26, the switches are set so that data may be circulated using the shift register clock lines 31. Line 31A clocks register 26A and line 31Z clocks register 26Z. As data is shifted out of the last position in register 26A, it is re-circulated to the first position in this register through switch 28A. By control of each shift register in this manner, all the data representing an entire line of super- pixels is now presented in a variety of groupings to module 33 which performs a comparison and correlation function. At any given moment, the particular grouping on which this function is performed comprises that data located in the last positions of registers 26, shown as register elements 32A through 32Z. The process whereby module 29 controls registers 26 is conveniently illustrated with the aid of Figure 4A, in which, for simplicity, a binary slice of the image data associated with object points A and B of Figure 2B is shown diagrammatically. Each square represents the value of unity for one perspective representation of a super-pixel on a background of zeroes. The data is arranged as part of a perspective array, horizontally by super-pixel position (the X-Axis), corresponding to distance across the object field, and vertically by perspective (the P-Axis). This arrangement of data is equivalent to that within the shift register array 26 of Figure 3 prior to the manipulation thereof by control module 29. The data associated with image point A" is spread across 14 super-pixels whereas that representing point B" requires 10 super-pixels, each point generating a locus of adjacent and corresponding data elements in the perspective array. Data elements A2 and B2 represent the right-most perspective P2 of, respectively, object points A and B, whereas elements A3 and B3 represent their left-most perspective P3. The process of image data compression now proceeds in three stages. First, the data is shifted by module 29 by a different amount for each shift register 26A through 26Z, so that the data associated with any object point in the nearest depth plane of interest will be arranged vertically (the same position in each shift register) in the register array. For the plane of point B in Figure 2C, this corresponds in Figure 4A to advancing the data corresponding to perspective P3 by 9 pixel increments relative to the data of P2 in X and the data corresponding to perspective PI (the central perspective) by half that amount. This skewed representation of the original data set is circulated through the registers by module 29, until all the data has been circulated past the end of the registers identified as elements 32A through 32Z. It will be apparent that the relative skew between P2 and P3 is directly related to the difference in depth between point B and a point on some median plane for which the data has no skew with respect to the perspective axis. Module 29 repeats this process for all depth planes of interest by setting up the appropriate degree of skew in each case, prior to re¬ circulating the data. Thus for point A the degree of skew relative to the perspective axis of the image data in the perspective array will have the opposite sign to that of point B. Module 29 provides the means for extracting from the shift registers data groups according to their respective degree of skew in the perspective array. The second stage of the compression process is to generate a signal indicative of the correlation within any particular grouping of data elements for all relevant skew (or depth) values and for every position along the scan line of the original 3-D image. This function is performed by correlator 33 as the various data groupings, as set up by control module 29, are presented at the outputs of register elements 32A through 32Z. Although, in the most general case this is done for every target depth (skew) value and at every super-pixel position, in practice, once a particular grouping has been assigned to a foreground point, its constituents need not normally be considered as candidates for points in the background. The output values generated by correlator 33 are held as an array in data storage means 34 in a format similar to that in which the range of perspective values of super- pixels is held in the perspective array. Instead of perspective being the axis orthogonal to X, a depth value (the Z-Axis) is represented. The output of correlator 33 represents the likelihood that each grouping of data elements, presented at its inputs, derives from a single object point. For the binary slice image, this operation can be a simple addition, but in the more general case, the smaller the standard deviation of the data values around their mean, the greater the probability of their having the same point as origin. In this case correlator 33 provides both a correlation term and the most likely compressed picture element value, usually the mean of the data values presented to it. An embodiment of correlator 33, having the potential of high speed operation is illustrated with reference to Figure 5. A series of digital to analog (D/A) converters 37A to 37Z are connected to the respective outputs 32A to 32Z of shift registers 26A to 26Z (Figure 3), in such manner that one of the colour components (typically 8 bits of data) of each of these perspective specific super-pixel values is converted to a voltage proportional thereto. When the corresponding voltage conversions are all complete, line 38 is used to transfer this series of voltages to the respective outputs of sample and hold circuits 39A to 39Z. A series of analog multipliers 40A to 40Z then provide the outputs representative of the squares of these signals. Summing amplifier 41 provides an output representative of the sum of the squares (Si) of the values provided at the inputs of D/A converters 37A to 37Z, whereas summing amplifier 42 provides the straight sum (S2) of these terms. Multiplier 43 then provides the square of this straight sum. The equality, (N-l)V = (SI - (S2) 2 /N) , (1) where N is the number of values included in the calculation, provides the variance, V, of these values. Thus, for a given value of N, the right hand side of equality (1) is inversely related to the likelihood that the group of colour component values, presented at the inputs of D/A converters 37A to 37Z, are all measures of that component of the light from the same original object point. Put in a different way, the probability that a particular group of measured values are all independent samples of the same normally distributed variable (the actual value subjected to a measurement error) is greatest when the standard deviation of these measurements (taken to be the square root of their variance) about their mean is a minimum. To complete the description of Figure 5, a difference amplifier 44 is provided with the output of summing amplifier 41 (SI) and that of multiplier 43, suitably scaled by resistor ladder Rl and R2, (S2) 2 /N . A digital representation of the variance related value given by equality (1) is provided through analog to digital (A/D) converter 45 and stored in the array of store 34 of Figure 3. In addition to storing values proportional to variance, store 34 is used to retain the mean value of every grouping of colour components, provided through A/D converter 46. It will be clear that, for an image comprised of a number of spectral components which need not be limited to three or, indeed, the visible spectrum of colour, the processing performed by the circuit of Figure 5 is carried out independently for each such component, and that, for the purpose of establishing the likelihood that any given set of measured values belongs to the same object point, the total variance is obtained by adding the variances for all spectral components. This can be achieved by duplicating hardware and providing an additional summing amplifier before A/D converter 45, or by processing spectral components sequentially and doing the addition thereafter, in, or at the input to, array store 34. Conversely, the mean value of each spectral component is typically held independently within the array, as one component of a possible picture element value. The third stage of the compression process is to reduce the data stored in array store 34 to a line of compressed picture elements, each having a simple picture element value (the intensity of one or more spectral components) and a depth value or co-ordinate. This is accomplished by passing all variance values through a comparator module 35 for each position in X. The most likely depth value of each picture element (the one associated with a minimum in variance) may be selected, or if necessary computed, and this, together with its picture element value, is passed to output register 36. Normally, the selection process is based on finding the absolute minimum of the variance for all the groupings analysed for a given position in X for the image data of a central perspective view (PI in Figure 4). Where background is partially obscured by the edge of a foreground subject, however, a local minimum may be selected, leading to a multi- valued result in Z (depth value) and corresponding picture element values for each depth. In other words, there will be special cases where a background point may be obscured for certain perspectives of the scene. A reduced signal will result from correlator 33, but this will typically still represent a confidence level sufficient to assign a depth value to correlated data. Where the scene has components which partially obscure others, there will typically be a requirement for multi-valued compressed picture elements for certain regions of X. In general, some longitudinal data compression of the background data will provide sufficient retention of image detail without the need for additional data storage or transmission capacity. Thus, in Figure 4B, the point B with an X co-ordinate X B and depth value or Z co-ordinate Z B represents the entire string (or locus) of data elements between elements B2 and B3 in Figure 4A. The X co-ordinate of B in Figure 4B corresponds to that of Bl in Figure 4A. Similarly, with a different depth value Z Λ , a compressed picture element A is found at X Λ and represents the entire locus of elements between A2 and A3. The compressed picture elements associated with each point along the scan line of the original 3-D image are now stored in register 36. If 8 bits of depth resolution and 24 bits of conventional pixel value data are needed for each compressed picture element, then this can be achieved with little more than a 33% increase of bandwidth when compared to an equivalent 2-D image. Whilst the embodiments of image data compression means illustrated in Figures 3 and 5 comprise a number of electronic hardware modules, the functions of one or more of these may be implemented in a programmable device such as a digital computer. Many of the operations required may be carried out using a highly parallel architecture such as an array processor or neural network. The selection of data groups according to skew in accordance with the embodiment of Figure 3 is sequential for each skew selected, having been accomplished by the setting up of the required skew in an array of shift registers. In an alternative arrangement, groups having all degrees of skew of interest may be accessed at the same time (or randomly) from a readable array of data registers extending in both axes of an X-P co-ordinate system, such as that of Figure 4A, and, in effect, containing a sufficient portion of the data comprising the perspective array. The use of variance and/or standard deviation from a mean value (and the finding of a local or global minimum in such quantity) to establish the most likely grouping of different image data elements to represent the different perspective views of the same object point, and, thus, to derive a measure of its range, has been described with reference to Figures 3 to 5. A variety of alternative algorithms, which provide a suitable estimate of the degree of match within a group of image data elements, may be applied in accordance with the current invention. In particular, an algorithm, which requires fewer computations than a direct calculation of variance for every depth at all points of the image, without introducing significant errors in the matching process, may be employed. Whereas the embodiment of the correlation circuit illustrated (Figure 5) receives values corresponding to a series of perspective views along a chosen line in the X-P co-ordinate system, simply illustrated in Figure 4A, such a circuit may be modified to select only candidate values, which have not already been grouped as compressed picture elements corresponding to points in a real (or synthetic) object space which lie closer to the image generating apparatus (or "viewing position") than the depth value for which the matching process is to be carried out. A commensurate adjustment must be made to the scaling factor applied to the output of multiplier 43 (Figure 3). Instead of a resistor ladder, a further multiplier may be used to achieve this. Suitable adjustment must also be made to the scaling constants of A/D converters 45 and 46. The embodiments of Figures 1, 2A and 2B illustrate the use of a segmented aperture in accordance with the invention. Specifically, in Figure 1, these segments are accessed sequentially and, whilst the "right-most" and "left-most" perspectives are referenced in the description relating thereto, it will be clear that such segments may, in practice, lie along a line having any convenient orientation in space relative to the subject matter. In fact, for edges of objects running horizontally, such as the edge of a table, it may be convenient to arrange the scan direction of aperture segment selection vertically. The correlation procedure employed in embodiments described has involved the progressive skewing of image data initially arranged in a perspective array having an X-axis and P-axis. The degree of skew relative to the P-axis of the locus in this two-dimensional perspective array of any image data derived from a single object point may be established by track following, providing the matching data which forms the locus may be clearly distinguished from the data surrounding it. More typically, however, the correlation of image data along a line in the perspective array having a specific skew with respect to the perspective axis is carried out. In such a procedure it is assumed that the perspective views of an object point are derived from viewpoints spaced in a direction parallel to the X-axis. More generally, other combinations of perspective viewpoints may be used in accordance with this invention and will require appropriately adapted correlation procedures. A good example of an alternative scheme is one in which viewpoints (or aperture segments) are arranged as an equi- spaced circular array. The consequence of such an arrangement is that the shift in the image plane between different projections (or perspectives) of the same object point will have the same magnitude for adjacent perspective pairs, but the direction of this "stereo shift" will vary from pair to pair, having the orientation of the line joining the two corresponding aperture segments. As in the embodiments illustrated, multiple two-dimensional perspective views of the subject matter from points around the circle provide candidate image points for correlation in accordance with the invention. In this example groupings of image data elements must be selected on the basis of a trial stereo shift for each element in its respective view from the nominal X-Y position of the compressed picture element for which a depth value is being tested. A three-dimensional perspective array is employed. This has X and Y axes for each perspective view and a P-axis as before. In one embodiment, instead of linear displacement, the P-axis now represents the angle in polar co-ordinates of the selection aperture with respect to the centre of the circular array. The locus in the perspective array corresponding to a given object point is a helix. The direction of this locus changes, but its angle or skew with respect to the P-axis is substantially determined by the range of the corresponding object point and, for an embodiment similar to that of Figure 1, how far the image of this point is positioned in front of or behind the CCD array or other image capture device. It follows that a perspective array having four dimensions may be used in certain embodiments. In such an arrangement there are two image axes (X and Y) and two perspective axes (P x and P- * -) . A system in which twelve perspective views of an object field are obtained simultaneously has been described. A system providing six views and using the same number of CCD elements would provide twice the resolution along the X-axis and still provide good correlation for many object fields. Whilst the two-dimensional perspective array illustrated with reference to Figure 4A shows the locus associated with a given object point to be essentially a straight line, this need not be the case. Any lens distortion, for example spheric aberration in image forming means such as lens 2 in Figure 1, will cause some non-linearity in the locus. This can be established and allowed for, as an additional perspective- dependent term in the skewing process, prior to correlation. The embodiments of Figures 1, 2A and 2B of the invention all utilise imaging systems which provide multiple views of the scene to be analysed, by selecting a portion of a larger system's aperture, either sequentially or in parallel, so as to distinguish each perspective view from any other. It will be clear to those versed in the art that an image recorded on film may comprise a range of perspectives of its subject matter in such a way that a perspective selection process may be applied to this recorded image data. A good example of this is provided by patent application PCT/GB/91/1920 (Street). Certain embodiments provide a source of three-dimensional image data in super-pixel form, suitable for processing in accordance with the principles of this invention, to allow depth extraction of features and image data compression. It may be necessary to re-arrange such three-dimensional image data into a format suitable for processing in accordance with this invention and, in particular, to select data associated with individual perspectives as part of the perspective separation process. It also follows, that any suitable source of sequential images, such as may be obtained with a conventional camera moved along a well defined trajectory (typically orthogonal to the direction of viewing) and, thus, providing a series of perspectives of an object field, may be used in accordance with the principles of this invention. In this case, it may be necessary to use one or more known points in the object field as three-dimensional reference co-ordinates to which to relate all other image points. Such a procedure will, inter alia, remove any image displacement caused by pointing instability of the camera as it is moved. Throughout this description, reference is made to depth values, with occasional reference to depth co-ordinates. It will be understood that a compressed picture element has a depth value which may be translated into the depth co-ordinate of the object point to which it relates. Conversely, whilst the depth value of an object point may be its depth co¬ ordinate in a particular co-ordinate system, nothing in this description is intended to limit the scope for alternative representations of depth for an object point. A useful and particular example of such a representation is one in which the depth value is reciprocally related to the actual range. Such a scale provides a finer scale for near field objects and a coarser one for the far field. Furthermore, the depth values computed from an image field will depend on the particular points in space which the image forming system "focuses" onto the image capture means, as it is these points which will have a minimum of skew (in the absolute sense) in the perspective array. In practice therefor, there will often be a need to map the depth value associated with an object point and its corresponding compressed picture element to an actual position or co-ordinate in real space. A useful application of this invention is in the field of robot vision, where it is necessary to identify objects and their location in three dimensions. The depth co-ordinate or range of an object is particularly important for collision avoidance and/or remote handling. The procedure is as follows. The object field of interest is converted into a field of compressed picture elements each with a depth value in accordance with the invention. Objects are generally identified by their boundaries in the compressed image field, at which there will be an abrupt change in the corresponding depth values to a more remote value. This discontinuity in depth values between neighbouring compressed picture elements provides a powerful means for isolating the nearer entity, to which one of the elements belongs, particularly as, unlike colour, contrast or lightness, its depth value does not vary with ambient lighting conditions or background scenery. This characteristic of edge detection may be usefully applied in the field of television, where a foreground subject must be separated from a background regardless of its content. At present, this is only conveniently done if the latter has a known and specific colour. A further application of this invention arises in the field of 3-D television. In this case, compatibility with 2-D conventional television may be achieved by treating the depth value as data additional to a conventionally encoded 2-D pixel field. Investigations into the human brain's response to different depth cues have revealed that quite a low bandwidth in this additional signal component will still yield a very realistic rendition of a full relief image. The storage of 3-D image data in compressed picture element form allows the use of a small number of views to fully encode the surface of any object. For a spherical object and four base points defined by the corners of a tetrahedron containing the sphere, four compressed 3-D pictures, one taken from each point, completely maps the surface of the object. More complex shapes with re-entrant features may require additional views. The uses of such a coding scheme include applications such as plastic surgery, the storage of dental records, the measurement of volumes of many kinds and the field of virtual reality. A picture in 3-D form is also more secure against unwanted replication than a 2-D equivalent.

Claims: CLAIMS 1. Apparatus for the compression of image information comprising three-dimensional image data input means for providing data derived from different perspectives of an object field; means for holding the data in a perspective array having at least two dimensions with respective axes, a first dimension representing distance across the object field and a second dimension representing perspective of the object field, each of a plurality of points in the object field providing a locus in the perspective array, said locus being skewed with respect to the axis of the perspective dimension to an extent dependent upon the depth of the point in the object field; and analyser means for analysing the data to produce respective depth values for the object points from the respective degrees of skew in the perspective array of the loci associated with those points. 2. Apparatus as claimed in Claim 1 in which the analyser means includes means for identifying correspondence between adjacent image data elements in the perspective array. 3. Apparatus as claimed in Claim 2 wherein the data represents a plurality of spectral components and said correspondence comprises spectral correspondence. 4. Apparatus as claimed in any preceding claim wherein the image data input means comprises perspective separation means. 5. Apparatus as claimed in Claim 4 wherein perspective separation means comprises image forming means; primary aperture means having substantial extent in at least one direction of perspective change; and image capture means. 6. Apparatus as claimed in Claim 5 wherein perspective separation means comprises a secondary aperture selection means intermediate the object field and image capture means selecting, in use, a portion of the image forming rays from the primary aperture means, whereby image forming rays from a substantial range of perspectives of the object field are captured sequentially. 7. Apparatus as claimed in Claim 6 wherein the secondary aperture selection means comprises an aperture substantially smaller than the primary aperture means, said smaller aperture being movable relative to said primary aperture. 8. Apparatus as claimed in Claim 6 wherein secondary aperture selection means comprises a shutter array having elements which are selectively operable. 9. Apparatus as claimed in Claim 5 wherein perspective separation means comprises an array of small convergent elements in close proximity to the image capture means, so that in use, image forming rays from different perspectives of a point in the object field impinge at respectively different positions on the image capture means. 10. Apparatus as claimed in any of Claims 5 to 9 in which image capture means comprises a CCD array. 11. Apparatus as claimed in any of Claims 5 to 9 in which image capture means comprises spectral separation means. 12. Apparatus as claimed in Claim 4 wherein perspective separation means comprises means for re-arranging three- dimensional image data to provide said perspective array. 13. Apparatus as claimed in any preceding claim wherein the analyser means comprises means for extracting data groups according to their respective degree of skew in the perspective array; correlation means for performing a correlation function on the data groups, the correlation means giving a likelihood function corresponding to the probability that the members of a data group represent the same object point; and means for deriving depth values for object points from the degree of skew associated with a maximum in the likelihood function. 14. Apparatus as claimed in Claim 13 wherein the means for extracting data groups comprises means for shifting data in a plurality of shift registers, said data comprising all or part of the perspective array and representing a corresponding plurality of perspectives of the object field; means for shifting the data in all registers substantially in synchronism; and skew selection means. 15. Apparatus as claimed in Claim 14 wherein skew selection means comprises means for shifting data in respective registers by respectively different amounts, whereby a degree of skew in said data is selected prior to shifting said data in synchronism; and means for transferring successive data groups having the selected skew to the correlation means from a location fixed in each of a plurality of said shift registers. 16. Apparatus as claimed in Claim 14 wherein skew selection means comprises means for extracting substantially simultaneously a plurality of data groups having a corresponding plurality of degrees of skew from a group of data elements extending in at least two dimensions of the perspective array. 17. Apparatus as claimed in any of Claims 13 to 16 wherein correlation means comprises means for converting data groups into analog signals; means for processing said signals to provide a signal corresponding to the likelihood function; and means for converting said function into suitable form for subsequent processing. 18. Apparatus as claimed in any of Claims 13 to 17 wherein the likelihood function substantially comprises the variance of the data within each data group. 19. Apparatus as claimed in any preceding claim wherein the means for holding data in a perspective array comprises space for the data associated with at least six perspectives of the object field. 20. Apparatus as claimed in any preceding claim wherein the means for holding data in a perspective array comprises space for the data associated with at least twelve perspectives of the object field. 21. A method for the compression of image information comprising providing three-dimensional image data derived from different perspectives of an object field; holding the data in a perspective array having at least two dimensions with respective axes, a first dimension representing distance across the object field and a second dimension representing perspective of the object field, each of a plurality of points in the object field providing a locus in the perspective array, said locus being skewed with respect to the axis of the perspective dimension to an extent dependent upon the depth of the point in the object field; and analysing the data to produce respective depth values for the object points from the respective degrees of skew in the perspective array of the loci associated with those points. 22. A method as claimed in Claim 21 wherein analysing said data comprises producing an array of respective compressed picture elements, each element comprising at least one depth value. 23. A method as claimed in Claim 22 wherein analysing said data comprises locating discontinuities in depth value between neighbouring elements within the array; and assigning each discontinuity to the boundary of an object within the object field.

Inventors: STREET GRAHAM STEWART BRANDON, STREET, GRAHAM, STEWART, BRANDON
Applicants: STREET GRAHAM S B, STREET, GRAHAM, STEWART, BRANDON
Application year: 1992
Main classification: H04N7
IPC codes: H04N7, H04N13