Abstract:
A graphical input and display system having a user interface for selecting object(s) to be operated on by the system includes input devices permitting a user to select and establish image object(s) received from various image input sources. A processor, connected to the system, receives requests for various image element selection operations and also accesses a memory structure. The system memory structure includes a user interaction module, which allows a user to select image elements without pre-selecting an element selection mode, as well as data memory.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS  
       [0001]    The following copending applications, Attorney Docket Number D/A1595, U.S. application Ser. No. XX/XXXXX, filed Mar. 22, 2002, titled “Method and System for Interpreting Imprecise Object Selection Paths”, Attorney Docket Number D/A1597, U.S. application Ser. No. XX/XXXXX, filed Mar. 22, 2002, titled “Method for Gestural Interpretation in a System for Selecting and Arranging Visible Material in Document Images”, and Attorney Docket Number D/A1598, U.S. application Ser. No. XX/XXXXX, filed Mar. 22, 2002, titled “System and Method for Editing Electronic Images”, are assigned to the same assignee of the present application. The entire disclosures of these copending applications are totally incorporated herein by reference in their entirety.  
       INCORPORATION BY REFERENCE  
       [0002]    The following U.S. patents are fully incorporated herein by reference: U.S. Pat. No. 5,485,565 to Saund et al. (“Gestural Indicators for Selecting Graphic Objects”); U.S. Pat. No. 5,513,309 to Meier et al. (“Graphic Editor User Interface for a Pointer-Based Computer System”); U.S. Pat. No. 5,523,775 to Capps (“Method for Selecting Objects on a Computer Display”); and U.S. Pat. No. 5,760,773 to Berman et al. (“Methods and Apparatus for Interacting with Data Objects Using Action Handles”).  
     
    
     
       BACKGROUND OF THE INVENTION  
         [0003]    This invention relates generally to user interfaces to computational devices, and more particularly to applications in which displayed objects are selected using a mouse or stylus. Two types of interactive drawing/sketching/editing applications are currently in use, both of which support creation of new image material, through draw operations, and selection and manipulation of existing material, through editing operations. The types of interactive applications are distinguished by the emphasis placed on “sketch” and “editing” operations. In an image “editing” program, selection and manipulation of image objects is the primary activity. Therefore, stylus or mouse interaction is designed primarily to interpret stylus input as selection gestures, and the default interpretation of mouse or stylus activity is selection of existing image objects for manipulation. Tools for drawing objects are provided by auxiliary command objects, usually menus.  
           [0004]    In a “sketch” program, however, the primary activity is the “draw” operation. To facilitate the sketching process, it is important for users to be able to quickly execute a series of markings such as handwritten or sketched strokes, without having to perform a menu initiation command at every stroke. These programs are designed such that draw operations can be the default interpretation of mouse or stylus activity. The disadvantage to this type of program is that when priority is placed on draw operations, selection operations become demoted and require explicit menu choices or button clicks to invoke a selection, which impedes the smooth flow of multiple selection and manipulation operations.  
           [0005]    Using these currently available tools, it can be very difficult to select and move, delete, or otherwise modify salient collections of markings at will. In particular, many user interfaces offer object selection by using the mouse to drag a rectangle enclosing target objects, draw a freeform path enclosing target objects, or define the vertices of a polygon enclosing target objects. There are no extant techniques for making all three of these selection methods available without the user specifying manually which one they intend to use. Additionally, the step of choosing among selection modes disrupts the flow of the task.  
           [0006]    U.S. Pat. No. 5,485,565 to Saund et al. titled “Gestural Indicators for Selecting Graphic Objects” discloses a graphical imaging system, in which the rough location, size and shape of objects in the image is summarized by a first characteristic descriptor, representing a parametric “pose” computed for each object. A second characteristic descriptor, for example a “gesture matching” function, is provided in order to select the single object, or the set of objects, that best comports with the user&#39;s selection gesture. When most closely matched, these key characteristic descriptors permit simple and natural user gestures to distinguish among a large set of graphic objects that may overlap both spatially. User gestures can be simple slashes passing through the object, or quick, coarse approximations of objects&#39; shapes.  
           [0007]    U.S. Pat. No. 5,513,309 to Meier et al. titled “Graphic Editor User Interface for a Pointer-Based Computer System” discloses a graphical editor arranged to permit the user to edit selected graphic objects by highlighting the objects and moving them with editing handles. A bounding box is also drawn about the selected portions of the object. In various aspects of the invention, the user is permitted to edit the object by executing specific actions, including resizing, duplicating, distorting and moving either the entire object or only selected portions. After any of the editing operations is performed, the display is updated to reflect changes made during the editing step.  
           [0008]    U.S. Pat. No. 5,523,775 (1996) to Capps titled “Method for Selecting Objects on a Computer Display” teaches a method for selecting objects on a screen of a pen-based computer system. The user&#39;s intention to enter the selection mode is indicated by the user&#39;s performing a special action with the stylus, such as holding the stylus in one place for a minimum amount of time, tapping the stylus immediately prior to and in the same location the highlighted gesture is to begin, or drawing a small circle. Each of these approaches is prone to user error. For example, a user may place the stylus down while thinking about the shape of the stroke to be drawn and unexpectedly be placed in a “highlight” (selection) mode. Similar problems occur with the “tap-and-a-half” and “small circle” methods described in Capps. Like other existing methods in this field, under the Capps invention the user is required to declare in advance of their stroke that the stroke is to be interpreted in terms of “highlight” (select) mode.  
           [0009]    U.S. Pat. No. 5,760,773 to Berman et al. titled “Methods and Apparatus for Interacting with Data Objects Using Action Handles” teaches a central processing unit coupled to a pointer control device such as a pen, stylus or mouse, that permits the user to selectively position a pointer and activate an action handle on a display associated with a data object. Activation of the action handle signals the central processing unit of selections associated with the data object. Tapping or clicking on the action handle causes display of a context menu containing at least one command that may be invoked with respect to the data object. Dragging the action handle indicates movement or dragging of the action handle and the corresponding data object for an operation associated with dragging such as drag-and-drop.  
           [0010]    The present invention offers a user interface technique permitting users to perform any of four types of selection method (click/tap, rectangle drag, path draw, and polygon draw) using a mouse or stylus without having to specify among these selection methods explicitly using a menu or other intermediate step.  
         SUMMARY OF THE INVENTION  
         [0011]    Briefly stated, and in accordance with one aspect of the present invention, there is disclosed herein a graphical input and display system having a user interface for selecting image object to be operated on by the system. The system includes input devices permitting a user to select and establish image object(s) received from various image input sources. A processor, connected to the system, receives requests for various image element selection operations and also accesses a memory structure. The system memory structure includes a user interaction module, which allows a user to select image elements without pre-selecting an element selection mode, as well as data memory.  
           [0012]    In another aspect of the invention, there is disclosed a method for utilizing a user interface on a graphical input and display system to select image elements to be operated on by the system without pre-selecting an element selection mode. After user input is received, the system determines the selection mode of the input selection and performs the selection operation.  
           [0013]    In yet another aspect of the invention, there is provided an article of manufacture in the form of a computer usable medium having computer readable program code embodied in the medium. When the program code is executed by the computer, the computer usable medium causes the computer to perform method steps for selecting image elements to be operated on by the system, without pre-selecting an element selection mode. The program readable code causes the computer to receive a user input selection, determine the selection mode of the input, and perform the selection operation.  
           [0014]    In another aspect of the invention, there is provided a memory for storing data for access by a program being executed on a computer for invoking object selection operations for selecting objects to be operated on by the system without pre-selecting an element selection mode. The memory includes a displayed objects data structure stored in the memory, a selected region data structure, a selection gesture data structure, and selected objects data structure.  
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0015]    The foregoing and other features of the instant invention will be apparent and easily understood from a further reading of the specification, claims and by reference to the accompanying drawings in which:  
         [0016]    [0016]FIG. 1 object selection using a mouse or stylus;  
         [0017]    [0017]FIG. 2 is a block diagram showing general components of a system that can edit images according to the invention;  
         [0018]    [0018]FIG. 3 is a flow chart showing steps in the click/tap and polygon draw selection method;  
         [0019]    [0019]FIG. 4 is a flow chart showing steps in the rectangle drag selection method;  
         [0020]    [0020]FIG. 5 illustrates a free-form path selection gesture;  
         [0021]    [0021]FIG. 6 illustrates a display during an ambiguous selection gesture;  
         [0022]    [0022]FIG. 7 is a flow chart showing steps in the path draw selection method; and  
         [0023]    [0023]FIG. 8 illustrates selected region highlighting according to the present invention. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0024]    Disclosed herein is a method and apparatus for object selection in mouse and stylus editing of electronic images. In the following description numerous specific details are set forth in order to provide a thorough understanding of the present invention. It would be apparent, however, to one skilled in the art to practice the invention without such specific details. In other instances, specific implementation details such as parsing techniques for extracting characters from a document image, have not been shown in detail in order not to unnecessarily obscure the present invention.  
         [0025]    As will become apparent in the description below, the present invention finds particular advantage in editing text and line art contained in an image. Documents which are faxed or which are copied on a digital copier typically involve images that contain primarily text and graphics. As described with respect to the prior art, it is common that in order to edit any of the text contained in the image, extraneous processing such as Optical Character Recognition (OCR) or the placement of image information into layers must be performed. As will become apparent, the present invention minimizes extraneous processing and provides added flexibility to defining both text and graphical image information so as to allow the editing of a wider range of textual and graphical data in an image.  
         [0026]    A number of standard event handling techniques are utilized by existing user interfaces, such as a mouse click, mouse release, or mouse drag. In a mouse drag event, the mouse button is depressed as the mouse is moved. A stylus may be used instead of a mouse, replacing mouse press and mouse release events with stylus touch and stylus lift events. These event handling techniques are illustrated in FIG. 1, in which discrete objects  110  are shown as they may be displayed on a display screen. A single mouse click or stylus tap at point  120  on discrete object “R” selects “R”. For displayed objects  130 , objects “B”, “R” and “P” are selected through being enclosed by freeform path  140 . In displayed objects  150 , “B”, “G”, “R” and “P” are selected through use of a rectangle drag as they are enclosed by rectangle  160 . In the case of discrete displayed objects  170 , a polygon enclosure selects objects “B”, “G”, “R” and the trapezoid. Existing user interfaces permit any or all of these selection methods, but require the user to specify which mode is to be used before performing and object selection operation. Typically, this may be done by clicking on an icon in a toolbar.  
         [0027]    Referring now to FIG. 2, system  200  includes processor  210 , connected to receive signals from image input device  220  and user input device  230 . Image input device  220  could be a scanner, a facsimile receiver or other image receiver, a camera, or other appropriate device or could be a part of local or remote memory that stores an image in digital form. User input device  230  could, for example, be a keyboard with a mouse. Processor  210  can also be connected to image output device  240 , such as a screen display, a printer, a facsimile transmitter or other image transmitter, or a part of local or remote memory that can store an image in digital form. Processor  210  is also connected to access program memory  250  and data memory  260 . Program memory  250  includes user interaction module  254 . Data memory  260  includes displayed objects data structure  262 , selected region data structure, selection gesture data structure  266 , tolerance tunnel data structure  268 , and selected objects data structure  270 .  
         [0028]    [0028]FIG. 3 illustrates example steps in the click/tap and polygon draw method for object selection. At step  310  the user performs a mouse press event, which could be a tap or mouse click. A determination is made at step  320  as to whether polygon input mode is to be engaged. If the polygon input mode is to be engaged, at step  330  the user may repeat the click or tap action. When the click or tap action is repeated, the processor selects the objects enclosed by a polygon at step  335 . In the event the click or tap action is not repeated, the processor drops the next polygon vertex at step  340 . Returning to step  320 , if the user elects not to use the polygon input mode, the processor determines whether the object indicator is located on a selectable object at step  350 . If the object indicator is located on a selectable object, the user may then select this object at step  360 . If the object indicator is not located on a selectable object, then the user decides whether to repeat the click or tap at step  370 . If the click or tap is repeated, the processor initiates the polygon input mode at step  380 . When the click or tap is not repeated at step  370 , then the processor initiates the path/rectangle input mode at step  390 .  
         [0029]    Referring now to FIG. 4, there is shown a flow chart illustrating the steps in the rectangle drag and freeform path methods of object selection. At step  410  a mouse drag event is initiated. The user determines whether to utilize the polygon input mode at step  420 . If the polygon input mode is selected, the processor adjusts the current polygon vertex at step  425 . If the polygon input mode is not selected, a decision is made at step  430  as to whether to select the path/rectangle input mode. When the path/rectangle input mode is not selected, the processor, at step  440 , determines whether an object has been selected. If an object has been selected, it is moved at step  445 . If an object has not been selected, the processor takes no action. Returning now to step  430 , if the path/rectangle input mode is selected, then an evaluation of the path is initiated at step  450 . If a clear rectangle selection gesture is provided, the processor displays the drag rectangle at step  460 . If the selection gesture is a clear freeform path, then the drag path is displayed at step  470 . In those cases in which the selection gesture is ambiguous, both the drag rectangle and drag path are displayed at step  480 .  
         [0030]    Freeform path selection gestures and enclosing rectangle selection gestures are both initiated by the same event, a single mouse click not occurring over a selectable object. In this state, the spatial characteristics of the gesture are analyzed to determine whether it has the characteristics of a freeform path, a rectangle, or either/both. Depending on this analysis, either the path, the rectangle, or both are displayed. This decision may be made, for example, as shown in FIG. 5, which illustrates a display screen  500 . In the case in which point  510  is the starting point of the selection gesture, point  520  is the point on the gesture thus far farthest from starting point  510 , and  530  is the current location point of the gesture (the current position of the mouse or stylus). If the distance between initial point  510  and point  530  is less than the distance between point  520  (the maximum point) and point  530 , then the selection gesture is judged to be a freeform path selection. If the distance between initial point  510  and point  530  is greater than the distance between point  520  (the maximum point) and point  530 , then the selection gesture is judged to be ambiguous.  
         [0031]    [0031]FIG. 6 illustrates the display during an ambiguous selection gesture. In this example, display screen  600  shows both a drag rectangle gesture  610  and a freeform selection gesture  620  selecting object  630 . Both drag rectangle gesture  610  and freeform selection gesture  620  remain visible on the display as the selection gesture is being drawn. Many users are able to disregard the displayed freeform path or drag rectangle when their intent is to select using the other type of gesture. When the mouse is released from freeform path/rectangle selection mode, the same criterion is used to determine whether the final gesture is regarded as a freeform path, or, if not, then as an enclosing rectangle, as described in FIG. 7. If the user wishes to select only the object(s) enclosed by rectangle  610 , the user releases the mouse button. If the user desires to include additional object(s), then the user continues drawing a freeform path. Until a selection is made, both freeform selection path  620  and rectangular selection path  610  remain visible on the display.  
         [0032]    In FIG. 7, when mouse release event  710  occurs, the user determines whether to evaluate if the gesture is a freeform path or an enclosing rectangle at step  720 . If a decision is made to evaluate the input mode, at step  730  the processor evaluates the path. In the case in which an enclosing rectangle is identified as the gesture, then at step  740  objects are selected by rectangle. If a freeform path is identified as the gesture, then at step  750  objects are selected by freeform path.  
         [0033]    In an alternate embodiment of the present invention, once a user has caused objects to be selected by dragging a freeform path, dragging a rectangle or other parametric shape, or by placing the vertices of a polygon, not only are these objects displayed in highlight form, but also the entire region enclosed by the selection gesture is indicated by a distinct highlight indicator. For example, in one embodiment illustrated in FIG. 8, discrete objects  830  are displayed. The objects “Red” and “O” are selected by freeform path  810 . These selected foreground image objects themselves are highlighted by causing background pixels  840  occurring within a fixed small distance of these objects&#39; foreground pixels to appear as a preselected color, for example, a semitransparent green. Then, the enclosure region  820  of the selection gesture may be highlighted in a lighter shade of green, or any other preselected color. This highlight function occurs at step  740  in FIG. 7 (“Select object(s) by rectangle”), step  750  in FIG. 7 (“Select object(s) by freeform path”), and at step  335  in FIG. 3 (“Select object(s) enclosed by polygon”).  
         [0034]    This embodiment advantageously presents the user with a clear indication of the system&#39;s interpretation of the selection gesture. Additionally, the user is assisted in visually locating all foreground mage objects that were selected by the selection gesture. Finally, the highlighted enclosure region itself becomes “active”, such that a mouse or stylus action anywhere in the enclosure region may cause move, pop-up menu, or other command operations. In the context of an advanced image editing system such as described in copending U.S. patent application Ser. No. XX/XXXXX (“System and Method for Editing Electronic Images”), user interaction is facilitated by removing the requirement for positioning the mouse or stylus directly over a selected foreground object in order to move or otherwise modify the collection of selected objects.  
         [0035]    While the present invention has been illustrated and described with reference to specific embodiments, further modification and improvements will occur to those skilled in the art. Although discussed with reference to text and line art, the operations illustrated herein apply equally well to any type of image object. Additionally, “code” as used herein, or “program” as used herein, is any plurality of binary values or any executable, interpreted or compiled code which can be used by a computer or execution device to perform a task. This code or program can be written in any one of several known computer languages. A “computer”, as used herein, can mean any device which stores, processes, routes, manipulates, or performs like operation on data. It is to be understood, therefore, that this invention is not limited to the particular forms illustrated and that it is intended in the appended claims to embrace all alternatives, modifications, and variations which do not depart from the spirit and scope of this invention.