A user-machine interface is disclosed that uses a line to connect and/or to intersect various objects displayed on a screen. A start of the line may identify a source, an end of the line may identify a sink, and line shapes and/or objects intersected by the line may identify commands.

BACKGROUND

User-machine interface has made great strides with the introduction of icons displayed on a screen and a mouse manipulating a pointer. As is well known, icons may be “grabbed” by pointing a mouse, click and hold, and moving the icon anywhere on a screen. For example, if the icon is moved into a “shredder” icon, the icon or the document represented by the icon may be deleted.

SUMMARY

A user-machine interface is disclosed that relies on the use of a line, having a start and an end, that connects various objects displayed on a screen. The line may start at a source object, extend to intersect other objects, and end at some desired location on the screen. A location of the start of the line may identify a specific portion of the source object as a source. If the line extends to intersect a command object, then the source may be used as input to a command corresponding to the command object for processing. If the line ends at a sink object, the processed source may be placed at a location indicated by the end of the line. If the end of the line cannot be associated with an object, then the processed data may simply be displayed at a location of the end of the line, for example.

The line may include various line shapes. For example, if the line starts by substantially surrounding a portion of the source object, that portion surrounded by the line is identified as the source to be processed by later commands. The line shape at the start of the line may also imply commands that should be performed on the source. For example, if the start crosses out a word of the source object in a particular direction such as from right to left, this line shape may command cutting the word from the source object.

The line may extend into a line shape that does not intersect with any object. Such line shapes may be interpreted as commands. For example, a simple loop may indicate the “insert” command, for example. Thus, if the line starts with a straight line crossing out a word in a source object, extends into a loop, and ends at a particular point in a sink object, then the word in the source object may be cut and inserted at a location indicated by the end of the line.

Objects may be one or more sources, sinks or commands. Command icons may be simply a line (a command bar) drawn across the screen, to separate the screen into two portions, for example. A line drawn starting from one portion of the screen, crossing the command bar into another portion of the screen may indicated copying (or cutting) text from the first portion of the screen into the second portion of the screen, for example. Source and sink objects may be of any kind and not limited to text. For example, source and sink objects may be network related such as email addresses, cell phone numbers, TV or radio stations, for example. Further, commands may also be non-text related such as voice synthesizers, video displays, headphones, etc.

In view of the above, instead of entering commands via keyboard or using a mouse, drawing of a substantially contiguous line to form various shapes and intersecting various objects provides an intuitive and simple method for communicating with a machine. Such a user-machine interface may be easily adaptable to any type of devices including xerographic devices such as copiers or printers with display command entry areas, hand-held units such as PDAs, cell phones, etc.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1shows an exemplary display screen100that displays objects102-110. For this example, objects102-106are source/sink objects and objects108and110are command objects. Source/sink objects102-106may represent documents/files/ports/network connection, etc., to or from which data may be input and/or output. Command objects108-110may represent actions that may be performed relative to source/sink objects102-106or data within them.

For example,FIG. 2shows a substantially contiguous line116that begins at object102, intersects object108and ends at object106. The substantially contiguous line116may be formed by ordinary means such as a pointer or a stylus using a touch sensitive screen, for example. These and other methods for forming a line may involve small gaps in the line. In any case, any gaps in the line must be of a small enough magnitude for the appropriate technology to distinguish one line from multiple lines. A dashed circle112is used to identify the start of line116, and a double dashed circle114is used to identify the end of line116. The dashed circle and double dashed circle112or114may not be actually displayed but are used here to indicate starts and ends of lines. The intersection of line116with object108may be crossing object108as shown inFIG. 2, or parameters may be set so that line116may only need to come within a certain set distance of object108to be interpreted as intending to include object108.

Line116represents a command to direct a machine to perform a function as represented by object108using object102or a portion of object102as input and placing the output in object106. Thus, instead of using a pointer and a mouse, for example, to select object102, and dragging object102to object108for a function to be performed, line116is actually drawn as a visible entity that connects source/sink objects with command objects.

FIG. 3shows another example where a line118includes a line shape120at the source end and connects two command objects108and110before reaching the sink object102. As shown inFIG. 3, line shape120may have a substantially closed perimeter surrounding a portion of object104to identify the source. It is immaterial whether the closed perimeter is completely closed. The purpose of the substantially closed perimeter is to unambiguously identify the source. Any shape sufficient for performing the identification function is acceptable.

The functions to be performed on the identified source is a combination of two functions as represented by command objects108and110. The results after performing functions associated with command objects108and110may be placed in the object102at a location identified by the end of line118. The commands108and110may be performed in the order from source to sink along line118. Other order schemes may be constructed such as resolving order based on types of command objects included by line118.

FIG. 4shows another example in which line122includes line shapes124,126and128. Similar to line shape120, line shape124may identify a source data, and, additionally, specify a function such as copy or cut, for example. Line122does not cross a command object relative to the source. Instead, line shape126is included and may be interpreted to be with a predefined function. Further, line shape128specifies not only a location where the processed results should be placed, but also a specific space within which the process results should fit by adjusting text reflow, size, etc., for example. Thus, using line shapes124,126, and128, the data in object104may be identified and processed via the function identified by line shape124and line shape126, and the result placed in object102at the location identified by line shape128. Multiple line shapes may also be included to specify additional commands. For example, if the source data is to be copied, put into bold font with underline, then two different line shapes, one for bold and one for underline, may be used instead of the one line shape126.

FIG. 5shows an example display screen200that includes a display section204, a command bar206and a display section208. Display sections204and208may be touch sensitive, for example, to facilitate use of a stylus. Other ways of drawing a line and line shapes are well known, such as, for example, using a mouse or keys of a keyboard other than touch sensitive screens. The command bar206is a type of command object and a command is “entered” by extending a line to cross command bar206.FIG. 5may represent an example in which a user may be reviewing documents210-214using a PDA or cell phone, for example, and decides to enter an ink comment222such as “good concept.” Then, the user may want to place the comment at a particular location of document210. Thus, as shown inFIG. 6, the user may use a stylus to draw line224starting at ink comment222and ending at a position that ink222comment should be placed in document210. Upon receiving line224, a command processor may cut ink comment222, size it based on the display size of document210and place the sized ink comment at the location indicated by the end of line224as shown inFIG. 7.

FIG. 8shows a line shape216at the end of line218that indicates how to place ink comment222.FIG. 9shows another example in which line220starts at ink comment222and ends in a substantially closed perimeter line shape226to specify where and how ink comment222is to be placed. As shown, ink comment222is reflowed and sized to fit into space substantially encompassed by line shape226.FIG. 10shows yet another example of a line shape230that identifies the word “example” as source data to be copied to a position below ink comment222. In this case, line shape230is interpreted as a “copy”. The command processor could be set to interpret line shape230as a “cut” if it crosses the word “example” from left to right instead of right to left, for example.FIG. 11shows an example of a substantially closed perimeter line shape232that encompasses a graph in document214and making a copy (or moving it) to display section208via line234.

FIG. 12shows a command bar306that includes fields308-314. Each of fields308-314may be defined to represent possible functions that may be either predetermined or set by the user directly or set using a profile, for example. WhileFIG. 11identifies fields308-314as labeled rectangles, these fields may be identified via color, icon or other convenient and meaningful symbols familiar to the user.

InFIG. 13, a line304starts at ink comment222, intersects fields310and314and ends at document212. Fields310and314may identify other accessible machines to which ink comment222may be transmitted. For example, fields310-314may represent colleagues of the user. Thus, the user forming line304transmits ink comment222to colleagues corresponding to fields310and314to communicate the approval. Colleagues corresponding to fields308and312do not receive ink command222, even if viewing document212. If colleagues corresponding to fields310and314are also viewing document213, then the ink comment may be placed in the colleagues copy of document212at the location indicated by the user.

While examples shown inFIG. 5-FIG.13show ink comment222in display section208, other types of objects may be placed there. For example,FIG. 14shows email402as a source object, and line404starting at email402, and ending at a desired location in display section204. Line404crossing command bar206may be interpreted as a command to receive email402and display its contents as a document213in display section204.FIG. 15shows an example in which the user may place line406starting at document215and ending at sink object408. Sink object408may be a “send” object that may be initialized to send email and/or facsimile via a network connection, for example, to addresses listed in an associated address file. Thus, line406may correspond to a command to send document215as email to addresses indicated in the address file. Additionally,FIG. 15shows another line412that starts at ink comment414and ends at sink object410that may represent an instant message session. Thus, line412commands sending ink comment414to instant message(s) partner associated with object410. The instant message session may automatically display message inputs so that the user need not explicitly “receive instant messages.

FIG. 16shows command bar206replaced by command bar306, and email402is opened and displayed in display section204. Field308may represent “self” or “personal” and the email is not sent to any other party. Other network type communications may also be possible such as receiving or sending audio files, video clips, TV or radio stations, Internet connections or communications with cell phones via text-to-voice or voice-to-text, etc.

As discussed in connection withFIGS. 1-4, command objects may be in any shape and not limited to command bars206and306as discussed in connection withFIGS. 5-16. For example,FIG. 17shows command objects506and508in shapes of familiar icons. Command object506may be a voice synthesis icon that performs text-to-voice conversion to enunciate text. Command object508may be a magnifier command that displays, in magnified fashion, the selected text.

As shown inFIG. 17, line504includes line shape502that identifies a particular paragraph in document212. Line504may end at command object506. In this case, the machine inputs the selected text into a voice synthesizer and generates audio output to “read” the selected text. If the user further extends line504with line portion510shown as a dashed line to end at command object508, the selected text may be also displayed in portion512of display screen200, for example, and the word currently enunciated may be highlighted as shown by highlight514. In this way, a user of a hand-held unit with limited display space and resolution may have a portion of a displayed document read audibly and, if desired, the portion of the document may be displayed for confirmation. A headphone icon may be used if audio is output to a headphone for privacy.

FIG. 18shows an exemplary block diagram of a machine600that may perform the functions in connection with entering commands via lines as discussed above. Machine600may include a command processor602, a memory604, a line shape discriminator606, a command discriminator608, an input/output controller610, a display controller612and a network interface614. These components602-614may be coupled together via one or more buses616.

WhileFIG. 17shows machine600configured in a bus architecture format, other architectures may be used as is well known to one of ordinary skill. Additionally, while separate blocks are used to illustrate various functional units, these blocks may be combined or further divided based on available technology. Hardware implementation such as application specific integrated circuits (ASCIs) or software implementations using either general or application specific processors such as DSPs may be used.

After power on, input/output controller602monitors whether a user has entered a line input. Input/output controller610may be fast enough to detect in real time a stylus or finger, for example, has contacted a touch screen and follows the movements of the stylus in real time to determine whether a line is being input and whether objects are being crossed. When input/output controller610determines that a line has ended, the entered information may be sent to line shape discriminator606to determine whether particular line shapes have been drawn. Line shapes may also be determined in real time. If line shapes have been drawn, line shape discriminator606may convert various line shapes into codes, for example, and the line shape codes sent to command discriminator608to determine what command the user has entered.

Command discriminator608may transmit the commands to command processor602which in turn may access information from memory604, process source data based on the commands, and output processed data into appropriate portions of memory604. If the user commands require access of network resources, command processor602may access such resources via network interface614. After the commands have been processed, command processor602may update display screen via display controller612to provide feedback to user of results of executed commands.

FIG. 19shows a flowchart700of an exemplary process for processing lines as discussed above. For convenience of discussion, flowchart700assumes that the process is fast enough to follow in real time a stylus using a touch screen to enter commands, as an example.

After power on, the process starts by going to step702and determines whether a line input has begun. If a line input has begun, the process goes to step704; otherwise, the process returns to step702. In step704, the process determines whether a line shape has been entered. If a line shape was entered, the process goes to step708; otherwise, the process goes to step706. In step706, the process identifies a source for data based on the location of the start of the line. For example, if the line starts at an input area such as display section208, the complete ink comment222may be determined to be the source. As discussed above, other types of icons may be used that may represent email, facsimile, instant messenger, etc., which may be selected by the start of the line.

In step708, the process identifies the source based on the particular line shape entered and whether a command is also included. For example, if the line shape substantially surrounds a paragraph of a document, then the complete paragraph may be identified as the source data. If the line shape is a substantially straight line crossing out particular portions of an underlying text, then such text may be either copied or cut as determined by various parameters that may be initialized.

After step706or708, the process goes to step710. In step710, the process determines whether a line shape is input that is not associated with a source/sink object. If such a line shape is entered, the process goes to step712; otherwise, the process goes to step716. In step712, the process determines the type of line shape entered and identifies a command associated with the entered line shape. If no command is associated with the line shape, then the process may assume that a command was not intended and goes to step716without identifying a command, for example. In step714, the process determines whether the line intersected an object. If the line intersected an object, the process goes to step715; otherwise, the process goes to step718. In step715, the process determines whether the intersected object is a command or a source/sink object. If the object is a sink object, then the process goes to step724; otherwise, the process goes to step716. In step716, the process identifies the command and queues the command for processing at an appropriate time and goes to step718.

In step718, the process determines whether the line has ended. If the line ended, the process goes to step720; otherwise, the process returns to step710. Thus, the process may accept multiple line shape/line-object sequences to perform complex functions. For example, the user may extend a line to surround particular text of an object to identify it as a source; then the line may further extend into another line shape that may be interpreted as a bold command so that the prior selected text is converted into bold font; then the line may further extend into another line shape that italicizes the selected text, and then the line may end at a sink object identifying a specific location so that the bolded and italicized selected text is placed at the identified location in the sink object. Thus, multiple commands may be selected to achieve combinations of operations that result in a complex process.

After step118, the process goes to step720. In step720, the process determines whether a line shape has been entered at the end portion of the line. If a line shape was entered, the process goes to step722; otherwise, the process goes to step724. In step722, the process identifies the command indicated by the line shape such as identifying a specific location to place the results of the commands. After step722, the process goes to step724. In step724, the process may perform the commands identified by the line. While flowchart700indicates that commands are performed in step724, depending on the type of command, the commands may be performed as they are identified or when all needed information is available. After step724, the process goes to step726. In step726, the process determines whether the process has been turned off. If the process is turned off, the process goes to step728and ends; otherwise, the process returns to step702to detect whether a new line entry has begun.

The above-described user-machine interface may be applied in any context where a display screen is used. For example, in xerographic machines such as printers or copiers, a command panel is often available and may use the line system described above. When PDAs or cell phones are used, touch sensitive screen technology may be used so that a stylus or a user's finger, a pencil, etc., may be used to draw lines for interface with these small types of machines.