Patent Publication Number: US-7724287-B2

Title: Sketch effect for digital photographs

Description:
BACKGROUND 
   Conventional film and more recently, digital cameras, are widely commercially available, ranging both in price and in operation from sophisticated single lens reflex (SLR) cameras used by professional photographers to inexpensive “point-and-shoot” cameras that nearly anyone can use with relative ease. Digital cameras are available with user interfaces that enable a user to select various camera features (e.g., ISO speed and red-eye removal). Some camera systems also allow the user to add creative effects (e.g., sepia tones, borders, etc.) to their photographs 
   Little is commercially available for allowing the user to create images from their photographs that appear as line drawings or “sketches” of the scene being photographed. Although photo-editing algorithms have been developed to create so-called “cartoon” effects, these are fairly sophisticated, requiring extensive processing power, and therefore are only suitable for use on computer systems. Digital camera users would need to download their photos to a personal computer (PC) to use the software before they can add such creative effects to their digital images. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
       FIG. 1  is a block diagram of an exemplary camera system which may implement a sketch effect for digital photographs. 
       FIG. 2  shows an exemplary digital photograph, a filtered image corresponding to the digital photograph, and a sketch image corresponding to the filtered image. 
       FIG. 3  is a simplified image illustrating an exemplary implementation for generating a sketch effect for a digital photograph. 
       FIG. 4  shows various exemplary sketch effects corresponding to the simplified image shown in  FIG. 3 . 
       FIG. 5  is a flowchart illustrating exemplary operations to implement a sketch effect for digital photographs. 
       FIG. 6  is a flowchart illustrating exemplary connected component labeling operations to implement a sketch effect for digital photographs. 
   

   DETAILED DESCRIPTION 
   Systems and methods are disclosed for creating a sketch effect for digital photographs. Exemplary systems may be implemented as an easy-to-use user interface displayed on the digital camera and navigated by the user with conventional camera controls (e.g., arrow buttons and zoom levers already provided on the camera). The user needs little, if any, knowledge about photo-editing, and does not need special software for their PC to create a sketch effect for their digital images. Various user options for creating the sketch effect are also available so that the desired sketch effect can be selected directly on the camera itself and can then be transferred to the user&#39;s PC (e.g., for sharing via email), printer, and/or photo processing station (e.g., Internet or store-based) to generate prints. 
   Exemplary Systems 
     FIG. 1  is a block diagram of an exemplary camera system which may implement a sketch effect for digital photographs. The exemplary camera system may be a digital camera  100  including a lens  110  positioned to focus light  120  reflected from one or more objects  122  in a scene  125  onto an image capture device or image sensor  130  when a shutter  135  is open (e.g., for image exposure). Exemplary lens  110  may be any suitable lens which focuses light  120  reflected from the scene  125  onto image sensor  130 . 
   Exemplary image sensor  130  may be implemented as a plurality of photosensitive cells, each of which builds-up or accumulates an electrical charge in response to exposure to light. The accumulated electrical charge for any given pixel is proportional to the intensity and duration of the light exposure. Exemplary image sensor  130  may include, but is not limited to, a charge-coupled device (CCD), or a complementary metal oxide semiconductor (CMOS) sensor. 
   Camera system  100  may also include image processing logic  140 . In digital cameras, the image processing logic  140  receives electrical signals from the image sensor  130  representative of the light  120  captured by the image sensor  130  during exposure to generate a digital image of the scene  125 . The digital image may be stored in the camera&#39;s memory  150  (e.g., a removable memory card). 
   Shutters, image sensors, memory, and image processing logic, such as those illustrated in  FIG. 1 , are well-understood in the camera and photography arts. These components may be readily provided for digital camera  100  by those having ordinary skill in the art after becoming familiar with the teachings herein, and therefore further description is not necessary. 
   Digital camera  100  may also include a photo-editing subsystem  160 . In an exemplary embodiment, photo-editing subsystem  160  is implemented in program code (e.g., firmware and/or software) residing in memory on the digital camera  100  and executable by a processor in the digital camera  100 , such as the memory and processor typically provided with commercially available digital cameras. The photo-editing subsystem  160  may include user interface logic  162  and effects logic  164 . 
   The effects logic  164  may be operatively associated with the memory  150  for accessing digital images (e.g., reading the images stored in memory  150  by image processing logic  140  or writing the images generated by the effects logic  164 ). Effects logic  164  may include program code for applying a sketch effect to the digital images stored on the camera  100 . The effects logic  164  may also be operatively associated with the user interface logic  162 . 
   User interface logic  162  may be operatively associated with a display  170  and one or more camera controls  175  already provided on many commercially available digital cameras. Such an embodiment reduces manufacturing costs (e.g., by not having to provide additional hardware for implementing the photo-editing subsystem  160 ), and enhances usability by not overwhelming the user with additional camera buttons. 
   During operation, the user interface logic  162  displays an effects menu on the digital camera (e.g., on display  170 ). In an exemplary embodiment, the effects menu may be accessed by a user selecting the design gallery menu option. The effects menu may then be navigated by a user making selections from any of a variety menus options. For example, the user interface logic  162  may receive input (e.g., via one or more of the camera controls  175 ) identifying user selection(s) from the effects menu. The effects logic  164  may then be implemented to apply a sketch effect to a digital image stored in the digital camera  100  (e.g., in memory  150 ) based on user selection(s) from the effects menu. 
   A preview image may be displayed on display  170  so that the user can see the sketch effect. Optionally, instructive text may also be displayed on display  170  for modifying, or accepting/rejecting the sketch effect. The instructive text may be displayed until the user operates a camera control  175  (e.g., presses a button on the digital camera  100 ). After the user operates a camera control  175 , the text may be removed so that the user can better see the preview image and sketch effect on display  170 . 
   Also optionally, the user may operate camera controls  175  (e.g., as indicated by the instructive text) to modify the sketch effect. For example, the user may press the left/right arrow buttons on the digital camera  100  to change the degree of line filtering, as explained in more detail below. 
   In an exemplary embodiment, a copy of the original digital photograph is used for adding a sketch effect to an image stored on the digital camera  100 . For example, the new image may be viewed by the user on display  170  directly after the original image so that the user can readily see both the original image and the modified image. 
   Before continuing, it is noted that the digital camera  100  shown and described above with reference to  FIG. 1  is merely exemplary of a camera which may implement a sketch effect for digital photographs. The systems and methods described herein, however, are not intended to be limited only to use with the digital camera  100 . Other embodiments of cameras and/or systems which may implement a sketch effect for digital photographs are also contemplated. 
     FIG. 2  shows an exemplary digital photograph  200 , a filtered image  210  corresponding to the digital photograph  200 , and a sketch image  220  corresponding to the filtered image  210 . The digital photograph  200  is first filtered, e.g., via a low pass filter into a secondary buffer to generate a gray-scale or black-and-white image of the digital photograph  200 . This filtered image  210  is then compared to the original digital photograph  200  on a pixel-by-pixel (or group of pixel to group of pixel) basis. Changes are rendered as lines in the filtered image  210 . 
   in an exemplary embodiment, each continuous line may be represented by a one byte per pixel rendering, e.g., to conserve memory resources in the camera system. This in effect turns the low pass filter into a high pass filter for the digital photograph  200 . 
   It is observed in  FIG. 2  that the filtered image  210  includes both acceptable outlines of objects from the digital photograph  200 , in addition to unacceptable lines or “noise.” Connected component labeling may then be applied to remove lines which do not satisfy a count threshold to reduce the number of lines (or noise) rendered in the sketch image  220 . Exemplary embodiments of connected component labeling can be better understood with reference to the simplified illustration described below with reference to  FIGS. 3 and 4 . 
     FIG. 3  is a simplified image  300  illustrating an exemplary implementation for generating a sketch effect for a digital photograph. For purposes of this illustration, a digital photograph has already been filtered to generate a gray-scale or black-and-white image of the digital photograph, and the filtered image has already been compared to the original digital photograph. Changes are rendered as lines in the filtered image  300  shown in  FIG. 3 . 
   The filtered image  300  includes continuous lines representing a circle object  310 , continuous lines representing a square object  312 , and continuous lines representing a triangle object  314 . It is noted, however, that the systems and methods described herein are not limited to use with simplified line data such as this (see, e.g.,  FIG. 2 ). 
   As discussed briefly above, connected component labeling may be applied to reduce the occurrence of noise. During connected component labeling, the image  300  is analyzed by scanning the pixels (illustrated by the pixels  320  in  FIG. 3 ), or groups of pixels, for line data. The pixels may be scanned right to left and top to bottom on a first pass, then left to right and bottom to top on a second pass. Of course other embodiments for scanning the pixels are also contemplated. 
   In an exemplary embodiment, pixels containing no line data are assigned a “0” and each continuous line segment is assigned its own unique identifier. For purposes of illustration, each pixel containing line data for the circle object  310  is assigned a “1” (e.g., pixels  330 ), each pixel containing line data for the square object  312  is assigned a “2” (e.g., pixels  332 ), and each pixel containing line data for the triangle object  314  is assigned a “3” (e.g., pixels  334 ). A matrix may be generated based on the line data. An exemplary matrix is shown in Table 1. 
   
     
       
         
             
           
             
               TABLE 1 
             
           
          
             
                 
             
             
               Exemplary Matrix 
             
          
         
         
             
             
             
          
             
                 
               Line ID 
               Pixel Count 
             
             
                 
                 
             
          
         
         
             
             
             
          
             
                 
               Line 1 
               1524 
             
             
                 
               Line 2 
               1362 
             
             
                 
               Line 3 
               984 
             
             
                 
                 
             
          
         
       
     
   
   The exemplary matrix shown in Table 1 includes a line ID for each continuous line in the image  300 . In this example, there are three continuous lines, Line 1 representing the circle object  310 , Line 2 representing the square object  312 , and Line 3 representing the triangle object  314 . The matrix also includes a pixel count for each line. For example, 1524 pixels in the image were assigned a “1”, 1362 pixels in the image were assigned a “2”, and 984 pixels in the image were assigned a “3”. A sketch image may be rendered with only the lines satisfying a count threshold, as described in more detail now with reference to  FIGS. 4   a - c.    
     FIG. 4  shows various exemplary sketch effects corresponding to the image  300  shown in  FIG. 3 . In a first example, the count threshold may be 1500. Accordingly, only the lines satisfying a count threshold of 1500 or greater are rendered in the sketch image. As shown above in Table 1, only the circle object satisfies this count threshold, and accordingly, only the circle object  410  is rendered in the sketch image  400 . In this example, the lines corresponding to the square object and the triangle object are considered noise, and therefore are not rendered in the sketch image  400 . 
   In a second example, the count threshold may be 1000. Accordingly, only the lines satisfying a count threshold of 1000 or greater are rendered in the sketch image. As shown above in Table 1, both the circle object and the square object satisfy this count threshold, and accordingly, only the circle object  410  and square object  412  are rendered in the sketch image  402 . In this example, the lines corresponding to the triangle object are considered noise, and therefore are not rendered in the sketch image  402 . 
   In a third example, the count threshold may be 500. Accordingly, only the lines satisfying a count threshold of 500 or greater are rendered in the sketch image. As shown above in Table 1, the circle object, the square object, and the triangle object all satisfy this count threshold, and accordingly, the circle object  410 , square object  412 , and triangle object  414  are all rendered in the sketch image  404 . None of the objects were considered noise. However, other lines (not shown) having a count threshold less than 500 would still be considered noise and would not be rendered in the sketch image  404 . 
   In an exemplary embodiment, a scalable threshold may be implemented. Accordingly, the user may select different sketch effects (e.g., having more lines or less lines). Although the user&#39;s selection may depend at least to some extent on the user&#39;s preferences, typically the user will want to select the image that removes “noisy” lines while leaving enough lines to sufficiently convey a “sketch” of the image. 
   It is noted that implementing connected component labeling to track lines (and optionally line size), reduces processing requirements and time to produce the sketch effect. Accordingly, the sketch effect can be readily implemented on an embedded system, such as the camera system  100  described above with reference to  FIG. 1 . In addition, tracking lines and/or line size also enables the sketch effect to be applied to any image, independent of image size (e.g., a 3 mega-pixel image, a 10 mega-pixel image, a thumbnail image, etc.). 
   It is also noted that the implementation of connected component labeling shown and described with reference to  FIGS. 3 and 4  is optimized for line rendering. In other embodiments, however, the filtered data may also be inverted such that lines are ignored and regions are retained (e.g., to produce an inverted sketch effect). 
   Exemplary Operations 
     FIGS. 5 and 6  are flowcharts illustrating exemplary operations which may be used to implement a sketch effect for digital photographs. The operations may be embodied as logic instructions on one or more computer-readable medium. When executed on a processor (e.g., in the camera), the logic instructions implement the described operations. In an exemplary embodiment, the components and connections depicted in the figures may be implemented. 
     FIG. 5  is a flowchart illustrating exemplary operations  500  to implement a sketch effect for digital photographs. In operation  510 , an original image is compared to a filtered image. For example, a digital photograph may be passed through a low pass filter into a secondary buffer to generate a gray-scale or black-and-white image, which can be compared to the original digital photograph on a pixel-by-pixel (or group of pixel to group of pixel) basis. 
   In operation  520 , changes in the filtered image are rendered as lines. In an exemplary embodiment, each line may be represented by a one byte per pixel rendering, e.g., to conserve memory resources on the camera system. This in effect turns the low pass filter into a high pass filter, and the filtered image includes acceptable outlines and unacceptable lines or “noise.” 
   In operation  530 , connected component labeling is applied to all of the lines (both acceptable lines and lines considered to be noise). The connected component labeling reduces the occurrence of noise, as explained in one exemplary embodiment below with reference to the exemplary operations  600  shown in  FIG. 6 . In operation  540 , a sketch image is rendered based on the connected component labeling in operation  530 . 
   Other operations, not shown, are also contemplated and will be readily apparent to those having ordinary skill in the art after becoming familiar with the teachings herein. For example, the effects logic (or other logic) may store a separate copy of the digital image before applying the sketch effect to the selected digital image. Accordingly, the user can revert back to the original digital image if the user decides that they do not like the sketch effect they have chosen without having to undo all of the changes. In another example, the connected component labeling operation  530  may be applied to remove various degrees of noise, wherein the user may select which level of noise reduction results in the best sketch image. 
     FIG. 6  is a flowchart illustrating exemplary connected component labeling operations  600  to implement a sketch effect for digital photographs, e.g., operation  530  in  FIG. 5 . In operation  610 , the filtered image is analyzed for line data. For example, pixel data may be scanned right to left and top to bottom on a first pass, then left to right and bottom to top on a second pass. In operation  620 , a matrix is generated using the line data. In operation  630 , a count for all line IDs or labels in the matrix is determined. 
   In operation  640 , a sketch image may be rendered with only the lines satisfying a count threshold. In an exemplary embodiment, a scalable threshold may be implemented. Accordingly, the user may select from different sketch effects (e.g., between those having more or less lines). 
   It is noted that the exemplary embodiments shown and described are provided for purposes of illustration and are not intended to be limiting. Still other embodiments for implementing a sketch effect for digital photographs are also contemplated.