Abstract:
An adaptive electronic zoom system is described which includes an electronic camera having an image sensor with pixels, a controller communicating with an address generator to select in response to an external input or feedback signals from the electronic camera a subset of pixels from the pixels of the image sensor, the subset of pixels defining a zoom area. The zoom area can be located at any position within the image sensor area. The zoom ratio/pan/tilt of the zoom area is automatically adjusted so as to maintain the location of the zoom area entirely within the image sensor area. The adaptive electronic zoom system is particularly suited for set-top boxes for video conferencing applications.

Description:
FIELD OF THE INVENTION 
     The invention relates to a method of providing electronic zoom and electronic pan and tilt capabilities in digital imaging applications, and more particularly to an electronic camera wherein the electronic zoom and electronic pan and tilt functions are seamlessly interfaced. 
     BACKGROUND OF THE INVENTION 
     Electronic cameras which can be moved mechanically to rotate about a vertical axis (pan), about a horizontal axis (tilt) and to change the magnification of the displayed imaged by changing the focal length of the camera lens (zoom) are known in the art. Since mechanical control of pan, tilt and zoom (“MPTZ”) is rather expensive, electronic cameras have been developed which exclusively utilize solid state components to pan, tilt, and zoom. An electronic pan, tilt, zoom (“EPTZ”) camera is small, fast, inexpensive, quiet, reliable, durable, easy to manufacture and easy to upgrade. If the requirements for resolution and image quality are modest, such as in most video conferencing applications, such cameras can have an imaging lens with a fixed focus and a stationary support, i.e., no mechanical movement of the camera body or of components within the camera. 
     In MPTZ cameras, panning, tilting and zooming are performed independently of each other. For example, if the camera is panned to the right as far as it can go, the lens is still free to zoom out and in to any position, from full wide angle to full telephoto, which can represent a factor of 25 to 30 in image magnification. On the other hand, in a stationary EPTZ camera having a fixed focus setting, panning, tilting and zooming operations are performed by selecting electronically a subset of pixels from all the pixels of the camera&#39;s image sensor. The image is zoomed in by selecting the subset of pixels from, for example, one fourth of the image area, corresponding to a zoom ratio of 2, and the image is panned/tilted by locating the subset of pixels in different areas on the image sensor. Tilting/panning are therefore not completely independent of each other, since an image can be panned/tilted only so far, until an edge of the subset of pixels of the zoomed image reached the edge of the image sensors. For example, if the camera is zoomed out to the widest possible view (full wide angle), then all pixels of the camera&#39;s image sensor contribute to the image to be displayed so that electronic panning or tilting is not possible since otherwise the subset of pixels would fall outside the pixel area of the image sensor. In a different situation, if the camera is zoomed in so that the subset of pixels is smaller than the pixel area of the image sensor and the image is subsequently electronically panned and/or tilted by forming the subset of pixels from pixels which are located closer to a lateral edge of the image sensor, the image can only be panned and/or tilted until an edge of the subset of pixels overlaps with the lateral edge of the image sensor. Alternately, if an image which is panned or tilted, is subsequently zoomed out, the image can only be zoomed out until an edge of the subset of pixels hits the lateral edge of the image sensor. 
     Conventional MPTZ camera adjustments are usually performed in a particular order. For example, starting at full wide view, a user would typically frame an object on, for example, the right side of the field of view by first panning the camera mechanically until the object is centered. The user would then zoom in until the object is framed with the desired magnification (zoom ratio). However, since an EPTZ camera in full wide view cannot pan or tilt, as described above, the user would first have to zoom in, with the possibility that the object moves out of the frame; the user would subsequently pan to the right, bringing the object back into the frame, then make final zoom adjustments, possibly switching back and forth between panning and zooming to bring the object close to a position which is centered in the displayed image. This process can be time consuming and frustrating. 
     It is therefore an object of the present invention to overcome these disadvantages by integrating in an EPTZ camera the zoom function with the pan and tilt function. 
     SUMMARY OF THE INVENTION 
     The object of the invention is solved by an electronic zoom system which includes an electronic camera with an image sensor with pixels, a controller communicating with an address generator to select a subset of pixels from the pixels of the image sensor, with the subset of pixels defining a zoom area. The zoom area can be located at any position within the image sensor area, with the defining pan and tilt of the zoom area. The zoom ratio/pan/tilt of the zoom area is automatically adjusted so as to maintain the location of the zoom area entirely within the image sensor area. 
     In an advantageous embodiment, the controller supplies data to the address generator to adjust the zoom ratio as a function of the respective pan and tilt position so that the zoom area is located entirely within the imager area. For example, if the image pans and/or tilts so far that a lateral edge of the zoom area abuts the lateral edge of the image sensor, then the image can continue to pan and/or tilt while at the same time, the image is zoomed in, i.e., the size of the subset of pixels is decreased, so that the lateral edge of the zoom area is prevented from crossing over the lateral edge of the image sensor. The zoom/pan/tilt operation is here completely linked and integrated so that the user or an automatic system, as mentioned above, will only have to select and electronically “aim” at the object to be imaged. 
     In another advantageous embodiment, the scaler automatically adjusts the pan and/or tilt position if a panned and/or tilted image is subsequently zoomed out (i.e., demagnified), so that the lateral edge of the zoom area is prevented from crossing over the lateral edge of the image sensor. 
     In yet another advantageous embodiment, the electronic camera is incorporated in a set-top box for video conferencing and has preferably an imaging lens with a fixed focal length. 
     In still another advantageous embodiment, the zoom system is responsive to characteristic image attributes, such as movement of an object, or environmental settings, such as sound or optical signals, e.g., a light beam emitted by a pointer. 
     In another advantageous embodiment, the electronic zoom/pan/tilt functions are incorporated in a scanner adapted to scan, for example, photographic images. A user can then use, for example, the control commands available on the user interface to zoom and pan/tilt the displayed image and the control electronics automatically adjusts, if necessary, at least one of the zoom ratio and the pan/tilt positions in response to changes in the other, so that no portion of the zoom area is located outside the image area. 
     The invention advantageously also provides a method for electronically zooming, panning and tilting an image. A zoom area is selected from an image area together with a pan and/or tilt position of the zoom area. When the displayed image is to be changed, a combination of the zoom area and the pan and/or tilt positions are adjusted so that no portion of the zoom area is located outside the pixel area of the imager. 
    
    
     BRIEF DESCRIPTION OF THE DRAWING 
     Other objects, features, and advantages of this invention will be described in the following detailed description with reference to preferred embodiments and the appended drawings, in which: 
     FIG. 1 is a front view of a set-top box incorporating an electronic zoom control of the invention; 
     FIG. 2 is a block diagram of an embodiment of an adaptive electronic zoom control in accordance with the invention; 
     FIGS. 3 a-c  illustrate a panned and tilted zoomed-in pixel area located within the pixel area of the image sensor, with the pan position from FIG. 2 a  to FIG. 2 c  increasingly shifting to the right, associated with an increase in the zoom ratio; 
     FIGS. 4 a-d  illustrate a panned and tilted zoomed-in pixel area located within the pixel area of the image sensor, with the zoom ratio decreasing from FIG. 3 a  to FIG. 3 d , associated with a shift in the pan and tilt position; 
     FIG. 5 is the minimum zoom ratio as a function of the pan/tilt position of the zoomed image; 
     FIG. 6 is a flow diagram for pan/tilt of a zoomed image according to the invention; and 
     FIG. 7 is a flow diagram for zooming a panned/tilted image according to the invention. 
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     Referring first to FIGS. 1 and 2, there is illustrated a set-top box  10  incorporating in a camera  20  the adaptive electronic zoom control system  90  of the invention. The camera  20  has preferably a lens with fixed focal length and, in the present embodiment, no mechanical movement, so that zooming, panning and tilting are performed entirely electronically. The set-top box  10  can also incorporate an optical receiver  22  for receiving an external optical input, for example an IR signal from a remote control unit (not shown) or a laser pointer (not shown). The set-top  10  box can also be enabled to receive acoustical input through a built-in microphone  30 , which is preferably directional. Remote microphone(s) can also be used (not shown). The camera  20  can then automatically electronically pan, tilt and zoom to capture an image of a speaker in response to signals received from the directional microphone  30  or from the respective remote microphone(s) positioned close to a speaker (not shown). It will be appreciated that the image sensor  110  itself can also be used instead of or in addition to input from the optical receiver  22  or the microphone(s) for identifying an object to be imaged. The camera  20  can also be directed to image an object (that is to automatically pan/tilt/zoom) by predetermined characteristic features of the image being acquired, such as distinct colors, movement, etc. 
     The set-top box  10  is preferably placed on top of a video monitor (not shown), with a video output signal received from the set-top box  10  connected to a monitor (not shown), for example, via a cable (not shown). 
     In FIG. 2, there is illustrated in the form of a block diagram an exemplary embodiment of an electronic zoom system  90  having electronic pan and tilt capability, with an electronic camera module  100 , an address generator  160  for selecting pixel addresses of a zoomed, panned and tilted portion of an image acquired by the electronic camera module  100  and a scaler  170  for scaling the zoomed, panned and tilted portion of an image so that it can be displayed in a conventional video format on a monitor (not shown). The camera module  100 , the address generator  160  and the scaler  170  are in communication with a controller  180 . In the present example, the address generator  160  is configured for user-controlled image magnification and repositioning of a zoomed image into different pan and tilt positions, as described below. The address generator  160  can also be controlled, for example, by signals from the image sensor itself, such as movement of the object to be imaged, or by environmental parameters, such as sounds and optical signals. A lens (not shown) focuses an image onto the active surface of an image sensor  110  which can be, e.g., a CCD type progressive scan imager chip. The image sensor  110  has an array of pixels arranged in the form of a matrix with, for example, 640 horizontal and 480 vertical pixels. The analog output of the image sensor  110  is processed conventionally with, for example, a correlated sampler (CDS)  120  to provide electrical signals which correspond to the illumination at any given pixel site of image sensor  110 , including color information. 
     From the foregoing, it will be readily understood by those skilled in the art, that the invention can also be incorporated in a scanner (not shown) instead of a camera, performing essentially identical functions. Because of the superior image resolution, most scanners employ a linear image sensor (not shown) which successively scans the original image object line by line in a manner known in the art. The signals derived from the scanned image are then rearranged to form the equivalent of an area image. Consequently, the present invention can be applied with both area and linear image sensors as well as with images transmitted and/or stored in electronic form, e.g., images transmitted in a network. 
     Likewise, images transmitted electronically can also be zoomed/panned/tilted with the present invention. The camera module  100  is then replaced with a suitable receiver (not shown) providing video outputs Y video  142 , C video  144 , and H/V sync  146  in a manner known in the art. The embodiment is not separately indicated in FIG.  1 . It will be appreciated by those skilled in the art that video signals other than Y/G, such as RGB, MPEG, etc., can be used interchangeably. 
     The analog signal is digitized in a high speed analog-to-digital converter (ADC)  124 , such as the model CXD2311AR from Sony Corporation with a typical resolution of 8 or 10 bits, and is fed to a digital signal processor (DSP)  130  such as the model SAA8110 from Philips. The DSP  130  converts the CCD signal to standard luminance Y video signals and chrominance C video signals and also adjusts picture contrast, brightness (that is, exposure), color balance, and imager setup based on instructions received from controller  180 . A timing and drive pulse generator  140 , for example, a model CXD1267AN or CXD2452R from Sony Corporation, is responsive to controller  180  and outputs the drive signals needed by the image sensor  110  and the DSP  130 , and sends cam clock pulses and horizontal and vertical sync pulses  146  to the address generator  160 . 
     The address generator  160  responds to a control signals from controller  180  electronically changing the zoom ratio and/or pan and/or tilt position by selecting a subset of pixels from the pixel array of the image sensor  110 . As mentioned above, this request can be initiated by a user or by a camera signal and/or by environmental parameters. 
     Referring now to FIG. 3 a , there is illustrated the complete image area  200  of the image sensor  110  comprising, for example, 640 pixels horizontally by 480 pixels vertically. The image sensor area  200  has a width w i  and a height h i , which can be expressed either in the form of linear dimensions or as the number of pixels, i.e., 640 and 480, respectively. The image sensor area  200  has a geometrical center P c . A subset of pixels  210  with a geometrical center Pz c  and a respective width w z  and height h z  is selected from the total number of pixels of the image sensor  110 , representing a zoomed-in, panned and tilted image, hereinafter referred to as the “zoomed image.” The zoom ratio can be expressed as R z =w i /w z =h i /h z . It is assumed that the aspect ratio of the image is left unchanged, i.e., h i /w i =h z /w z , without limiting the scope of the invention. The pan position is (Pz c −P c ) horizontal  and the tilt position is (Pz c −P c ) vertical . In the present example, the zoomed image  210  of FIG. 3 a  is located completely within the image sensor area  200  so that the zoomed image  210  can be further zoomed in, panned and tilted. In the present example, with R z =2 and (Pz c −P c ) horizontal =100 pixels horizontally and (Pz c −P c ) vertical =−50 pixels vertically, the subset of pixels of the zoomed image  210  represents the pixels between 260 and 580 horizontally and between 70 and 310 vertically, placing the origin (pixel  1 ,  1 ) at the lower left corner of the complete image area  200 . The go address generator  160  generates the pixel addresses for the subset of pixels. 
     The zoomed image  220  of FIG. 3 b  is generated by electronically panning the zoomed image  210  of FIG. 3 a  farther to the right until the right vertical boundary of the zoomed image  220  coincides with the right vertical boundary of the image sensor  200 . The center of the new pan position is denoted as Pz′ c . The zoom ratio is not changed. The tilt is also left unchanged to simplify the discussion. In this case, (Pz′ c −P c ) horizontal =480 pixels horizontally and (Pz′ c −P c ) vertical =−50 pixels vertically, so that the subset of pixels of the zoomed image  220  represents the pixels between 320 and 640 horizontally and between 70 and 310 vertically. These addresses are then generated by the address generator  160 . 
     The zoomed image  220  cannot be panned farther to the right without adjusting the zoom ratio, since the subset of pixels would then require pixels located to the right of the vertical boundary of the image sensor  200 . According to the present invention, the zoom ratio is therefore automatically adjusted so as to keep the zoomed image within the boundaries of the image sensor  200 . This situation is illustrated in FIG. 3 c.    
     In FIG. 3 c , it is assumed that the user desires to pan the image farther to the right to a new pan position (Pz″ c −P c ) horizontal =540 pixels while the tilt position remains at (Pz″ c −P c ) vertical =−50 pixels. Since the zoomed image is already at the right edge of the image area  200 , the system, according to the invention, automatically and continuously increases the zoom ratio R z  from R z =2 to R z =3.2 to keep the right vertical boundary of the zoomed image  230  at the right vertical boundary of the image sensor  200 . In this case, the subset of pixels of the final zoomed image  230  represents the pixels between 440 and 640 horizontally and between 115 and 265 vertically. The respective pixel addresses of this new subset of pixels are then generated by the address generator  160 . It is understood that the allowable pan and/or tilt of the system is limited by the largest allowed zoom ratio which is determined, for example, by the desired image quality and image resolution. 
     The Y video signal  162 , the C video signal  164  as well as the addresses  166  of the zoomed images  210 ,  220  and  230 , respectively, are then passed on to the scaler  170  which “scales” the pixel count of zoomed images (i.e., 320×240 for images  210  and  220  and 200×150 for image  230 ) to the CIF format of 352×288 pixels for electronic transmission, to full video format of, for example, 640×480 pixels or to any other format desired for electronic manipulation, display, transmission or storage of the images by computing interpolated or decimated pixels values in a manner known in the art. The scaled Y video signal  172 , the scaled C video signal  174  as well as the scaled addresses  176  of the full video image are supplied by scaler  170  to a display device (not shown). 
     In another aspect of the invention which is illustrated in FIGS. 4 a - 4   d , the zoomed-in image  310  of FIG. 4 a  (which in the present example corresponds to the image  210  of FIG. 3 a ) is zoomed-out while initially leaving the pan/tilt position unchanged. The zoomed image  310  of FIG. 4 a  is located completely within the image sensor area  200  so that the zoomed image  310  can be at least initially zoomed out. In the present example, with R z =2 and (Pz c −P c ) horizontal =100 pixels horizontally and (Pz c −P c ) vertical =−50 pixels vertically, the subset of pixels of the zoomed image  310  represents the pixels between 260 and 580 horizontally and between 70 and 310 vertically. These pixel addresses are, as before, generated by the address generator  160 . 
     The zoomed image  320  of FIG. 4 b  is generated by zooming the zoomed image  310  of FIG. 4 a  out until, in this example, the right vertical boundary of the zoomed image  320  coincides with the right vertical boundary of the image sensor  200 . The pan and tilt position Pz c  is not changed. In the present example, a minimum zoom ratio R z  of 1.45 can be attained without adjusting the pan and tilt position Pz c . At that point, the right vertical boundary of the zoomed image  320  coincides with the right vertical boundary of the image sensor  200 . The subset of pixels of the zoomed image  320  represents the pixels between 200 and 640 horizontally and between 25 and 355 vertically. These pixel addresses are, as before, generated by the address generator  160 . 
     If the image  320  is electronically zoomed out further and reaches a vertical boundary of the image area  200 , then both the pan and the tilt positions Pz c  have to be adjusted automatically according to the invention in order to satisfy equation (2) given below. Likewise, the pan and the tilt positions Pz c  have to be adjusted if the zoomed-out image  320  reaches a horizontal boundary of the image area  200 . The image  330  illustrated in FIG. 4 c  is obtained from the image  320  of FIG. 4 b  by automatically maintaining the right vertical boundary of the zoomed image at the right vertical boundary of the image area  200  while the zoom ratio is decreased. In the example of FIG. 4 c , both the right vertical boundary and the lower horizontal boundary of the zoomed image  330  coincide with the respective right vertical boundary and the lower horizontal boundary of the image area  200 . The center (pan and the tilt position Pz c ″) of the zoomed image  330  is moved upwardly and to the left relative to the center Pz c  of image  320 . 
     In the illustrated example, the zoomed image  330  has a zoom ratio R z =1.26 as calculated from the original tilt position (Pz c −P c ) vertical =−50 pixels. Accordingly, the zoomed pixel  330  is 640/R z =506 pixels wide, yielding a new pan position Pz″ c =0.5 *(640−506)=67. The corresponding pixel addresses are again generated by the address generator  160 . 
     If the zoomed image  330  is electronically zoomed out farther, as indicated by zoomed image  340  in FIG. 4 d , both the pan and tilt position of Pz′″ c  are adjusted automatically and simultaneously with the zoom adjustment so as to keep the zoomed image  340  within the image sensor area  200 . The new pan and tilt position Pz′″ c  is preferably calculated by the controller  180  in accordance with the examples described above and with reference to FIG.  4 . 
     FIG. 5 illustrates the functional dependence of the minimum zoom ratio R z  permitted for a given relative pan position (Pz c −P c )/(w i /2) in the horizontal direction and a given relative tilt position (Pz c −P c )/(h i /2) in the vertical direction. It is evident from the foregoing discussion that the maximum value of the relative pan and tilt positions determines the minimum allowable zoom ratio. As mentioned above, the zoom ratio is expected to be limited to a largest value which is determined by the desired image quality. The curve of FIG. 5 can be described by the equation 
     
       
         |( Pz   c   −P   c )/( w   i /2)|=1−1 /R   z  and 
       
     
     
       
         |( Pz   c   −P   c )/( h   i /2)|=1−1 /R   z , respectively.  (Eq. 1) 
       
     
     Consequently, the minimum allowable zoom ratio R z  is: 
     
       
           R   z &gt;max(|( w   i /2)/( w   i /2−( Pz   c   −P   c ))|, 
       
     
     
       
         |( h   i /2)/( h   i /2−( Pz   c   −P   c ))|)  (Eq. 2) 
       
     
     The allowable zoom ratio R z i s thus located in the region of FIG. 5 which is bounded by the two curves  402  and  404  and by the maximum permissible zoom ratio R z,max  as determined by the line  406  and representing the desired image quality, as discussed above. The zoom system of the invention is designed so as to continually monitor and, if necessary, adjust the zoom ratio in response to the pan and/or tilt position and vice versa, so that the zoomed image remains within the pixel area of the image sensor  110 , as expressed by Eq. 2. 
     When the final zoomed/panned/tilted image selected by the user is attained, the Y video signal, C video signal and the horizontal and vertical pixel addresses of the zoomed image are routed via the respective signal paths  162 ,  164  and  166  to the scaler  170  as illustrated in FIG.  2 . Since the zoomed image has less pixels than the full image to be displayed by a monitor (not shown), as mentioned above, new interpolated pixel values are computed by the scaler  170  to provide the full complement of pixels values to be displayed, for example, 640×480 pixels. Additional image processing functions, such as, for example sharpening, can also be performed by the scaler. The scaled image signals are available as Y video, C video and horizontal and vertical pixel addresses at respective scaler outputs  172 ,  174  and  176 . 
     Referring now to FIGS. 6 and 7, there are shown flow diagrams of a preferred method for carrying out the invention. The flow diagram of FIG. 6 relates to a situation where an existing zoomed image  600  with an initial zoom ratio, pan and tilt position is panned and/or tilted electronically. As mentioned above, an image with a zoom ratio of one cannot be panned or tilted. In step  610  a new pan/tilt position is provided, either by a user or automatically in response to camera or environmental signals, as discussed above. In step  620 , the image is panned/tilted by Δ towards the new pan/tilt position. When panning/tilting continuously, the Δ position and the new position can be identical. Step  630  then checks if the panned/tilted image at the Δ position is inside the imager area. If this is the case, the next step  660  checks if the new pan/tilt position is reached. If this is the case, the final image is produced in step  670 . Otherwise, the process loops back to step  620  where the pan/tilt position is adjusted once more by Δ. 
     If the image or a portion thereof produced with the pan/tilt position in step  630  is outside the image area, then in step  640  the image is zoomed in according to the zoom ratio calculated for the respective pan/tilt position from Eq. (2). If, as determined in step  650 , the zoom ratio R z  is not greater than or equal to the maximum zoom ratio R z,max , then the process continues in step  660 , discussed above. If, as determined in step  650 , the zoom ratio R z  is greater than or equal to the maximum zoom ratio R z,max , then in step  655  the zoom ratio R z  is set equal to the maximum zoom ratio R z,max  and the pan/tilt motion is reversed to a position corresponding to R z,max  The final image is produced in step  670 . 
     The flow diagram of FIG. 7 is similar to that of FIG.  6  and relates to a situation where an existing image  600  with an initial zoom ratio, pan and tilt position is electronically zoomed out, that is, R z  decreases. It is evident that an existing image  700  which is located entirely within the image sensor area  200 , can always be zoomed in. In step  710 , a new zoom ratio is provided, either by a user or automatically in response to camera or environmental signals, as discussed above. In step  720 , the image is zoomed out by ΔR z  towards the new zoom ratio R z . When zooming continuously, the zoom ratio adjusted by ΔR z  and the new zoom ratio can be identical. Step  730  then checks if the zoomed-out image at the ΔR z  zoom ratio is inside the imager area. If this is the case, the next step  760  checked if the new zoom ratio is reached. If this is the case, the final image is produced in step  770 . Otherwise, the process loops back to step  720  where the zoom ratio is adjusted once more by ΔR z . 
     If the zoomed-out image produced with the adjusted zoom ratio in step  730  or a portion thereof is outside the imager area, then in step  740  the image is panned/tilted according to the pan/tilt position calculated for the respective zoom ratio from Eq. (2). If, as determined in step  750 , the zoom ratio R z  is not less than or equal to one, then the process continues in step  760 , discussed above. If, as determined in step  750 , the zoom ratio R z  is less than or equal to one, then in step  765  the zoom ratio R z  is set equal to one and the pan/tilt position is set to zero. The final image is produced in step  770 . 
     The foregoing is considered to be illustrative only of the principles of the invention. Modifications of the preferred embodiments disclosed herein will be apparent to those skilled in the art that are within the scope of the following claims.