Reversible user interface component

A method of configuring a widget and a tactile user interface which displays the widget are disclosed which enable menu-setting operations with minimal touch gestures and occupation of screen space. The interface includes a touch sensitive display device and memory which stores instructions for displaying the widget and a set of graphic objects together on the display device. The widget has two or more virtual sides, each of the sides being associated with a respective functionality. The widget is flipped, in response to a recognized touch gesture, from a first of the sides to a second of the sides, whereby the functionality of the widget is changed. The graphic objects are associated, in memory, with respective items having attributes. The graphic objects exhibit a response to the widget functionality of a currently displayed one of the sides of the widget based on the attributes of the respective items.

BACKGROUND

The exemplary embodiment relates to tactile user interfaces and finds particular application in connection with a displayable user interface component (“a widget”) having a plurality of virtual sides, whereby at any given time, only one side is displayed to the user on the user interface, and the other side or sides are hidden from view, each side being associated with a different functionality.

Multi-touch interactive systems using specific user-interface designs and capabilities allow users to navigate easily through interactive content on multi-touchscreens of interactive table computers, interactive windows, multi-touch tablet computers, and multi-touch tablet PCs, and smartphone devices, all of which are referred to herein as tactile user interfaces (TUIs). TUIs incorporate a display device and touch-screen which detects user hand or finger movements or movements of another implement with which a user contacts the screen, such as a stylus. The detected movements are translated into commands to be performed, in a similar manner to conventional user interfaces which employ keyboards, cursor control devices, and the like. Such tactile user interfaces can be used for manipulating graphic objects which can represent underlying documents, through finger or implement-guided movement of a widget, such as a virtual magnet.

However, translating the design of standard graphical user interfaces to multi-touch interactive devices is not always straightforward. This can lead to complex manipulations that the user may need to memorize in order to use the functionality provided by a touch-screen application. Additionally, hand movements often lack the precision which can be achieved with a keyboard and user fingers differ in size and shape, causing different touch signals to be sent from the touch-screen to the application.

When a user wishes to change functionality of the widget, this may be performed using a menu from which a user selects one of the functionalities. Such a menu is described, for example, in copending application Ser. No. 13/031,336 and in U.S. Pub 20100313124. As disclosed therein, the functionality of a displayed virtual magnet is changed by selecting from a menu displayed, for example, on a ring around the virtual magnet, with navigable sub-menus optionally being accessed from a main menu. The magnet can then be applied to the set of graphic objects where the new functionality of the magnet affects how the graphic objects respond to the magnet, based on underlying document properties.

One problem with such a widget is that on some user interfaces, such as smartphones, the menu options may be difficult to read, due to the small size of the screen. Selection of one of the menu options using touch gestures may also be difficult or time consuming, even on larger screens, when the number of options is fairly large or when navigation through a number of sub-menus is desired.

There remains a need for an apparatus and method employing a displayable widget which can be tuned with minimal touch gestures.

INCORPORATION BY REFERENCE

The following references, the disclosures of which are incorporated herein by reference, in their entireties, are mentioned.

U.S. application Ser. No. 13/031,336, filed Feb. 21, 2011, entitled QUERY GENERATION FROM DISPLAYED TEXT DOCUMENTS USING VIRTUAL MAGNETS, by Caroline Privault, et al.

U.S. application Ser. No. 12/976,196, filed on Dec. 22, 2010, entitled SYSTEM AND METHOD FOR COLLABORATIVE GRAPHICAL SEARCHING WITH TANGIBLE QUERY OBJECTS ON A MULTI-TOUCH TABLE, by Yves Hoppenot, et al.

U.S. Pub. No. 20110282892, published Nov. 17, 2011, entitled METHOD AND SYSTEM TO GUIDE FORMULATION OF QUESTIONS FOR DIGITAL INVESTIGATION ACTIVITIES, by Stefania Castellani, et al.

U.S. Pub. No. 20090077501, published Mar. 19, 2009, entitled METHOD AND APPARATUS FOR SELECTING AN OBJECT WITHIN A USER INTERFACE BY PERFORMING A GESTURE, and U.S. Pub. No. 20090128505, published May 21, 2009, entitled LINK TARGET ACCURACY IN TOUCH-SCREEN MOBILE DEVICES BY LAYOUT ADJUSTMENT, by Kurt E. Partridge, et al.

U.S. Pub. No. 20090100343, published Apr. 16, 2009, entitled METHOD AND SYSTEM FOR MANAGING OBJECTS IN A DISPLAY ENVIRONMENT, by Gene M. Lee.

BRIEF DESCRIPTION

In accordance with one aspect of the exemplary embodiment, a tactile user interface includes a touch sensitive display device. Memory stores instructions configured for displaying a widget and a set of graphic objects on the display device. The widget has a plurality of virtual sides, each of the sides being associated with a respective functionality. The widget is flipped, in response to a recognized touch gesture, from a first of the sides to a second of the sides, whereby the functionality of the widget is changed from a first functionality associated with the first side to a second functionality associated with the second side. The graphic objects are each associated, in memory, with a respective item having attributes. The graphic objects exhibit a response to the widget functionality of a currently displayed one of the sides of the widget based on the attributes.

In accordance with another aspect, a method for selection of a functionality for a widget displayed on a tactile user interface includes displaying a widget and a set of graphic objects contemporaneously on a display device of a tactile user interface, the widget having a plurality of virtual sides, each of the sides being associated with a respective functionality. The method further includes recognizing a touch gesture on the widget corresponding to a transformation of the widget from a first of the sides to a second of the sides and displaying the widget being flipped, in response to the recognized touch gesture, from the first of the sides to the second of the sides, whereby the functionality of the widget is changed from a first functionality associated with the first side to a second functionality associated with the second side.

In accordance with another aspect, a reconfigurable widget includes a first virtual side which is associated in memory of a computing device with a first functionality selected from a predetermined set of functionalities. Each of the functionalities is operable on displayed objects. The first functionality is actuated only when the first side is displayed on a touch sensitive display device and the second side is hidden from view. A second first virtual side of the widget is associated with a second functionality selected from the predetermined set of functionalities, the second functionality being actuated only when the second side is displayed on the display device and the first side is hidden from view. The widget is rotatable about an axis between the first and second sides in response to a recognized touch gesture on the display device.

DETAILED DESCRIPTION

With reference toFIG. 1, a tactile user interface (TUI)1is shown. The TUI includes a display device10, which includes a screen12, such as an LCD screen, and a touch sensitive surface14overlying the screen, which together may be referred to as a touch-screen. Other touch-screen devices are also contemplated.

A widget16is displayed on the screen12. The widget can have the appearance of an icon, such as a button, which is visible to a user of the TUI1. The exemplary widget16includes an arrangement of colored pixels displayed on the screen12. The widget16has a plurality of virtual sides18,20, etc. (seeFIG. 2), each side being associated with a different functionality. Each of the widget's functionalities is operable on contemporaneously displayed graphic objects22,24, as described below. Only one of the functionalities is capable of being “ON” (operational) at any given time, which corresponds to the virtual side which is displayed on the screen (“side1” inFIG. 1). One or more hidden (at least partially or completely obscured from view) virtual sides of the widget16have their functionalities “OFF”. The virtual sides18,20of the widget can thus together serve as a menu of options (functionalities), with a current chosen single one of the options being displayed to the user. In addition to being able to be translated between sides18,20, the widget16is moveable across the screen12, by means of touch gestures.

The exemplary TUI1can be configured as for the TUI disclosed in above-mentioned application Ser. No. 13/031,336 and/or Pub. No. 20100313124, incorporated herein by reference, except as noted. The user interacts with the TUI1primarily through finger contacts and gestures on the touch-sensitive touch-screen12,14. The interaction can be through one or more finger touches and/or through a large touch area in contact with the screen, such as single fist or hand palm, or any other object striking the screen, such as a stylus. The touch-screen12,14and associated software detects the user contacts and translation of their positions on the screen12to associate them with the widget(s)16and commands.

The exemplary widget16provides users with an alternative method for menu selection to the usual menu navigation and selection methods. The exemplary system provides fluid and fast changing of widget functionality at any time during the course of a user session on the TUI with a small number of gestures. In addition, it is particularly suited for small touch-screen devices, such as smartphones, where the size of the widget16can be reduced, while at the same time providing more menu options. Widget setting on the exemplary multi-touch display device10is intuitive and easy to learn for the user. The user can switch comfortably from manipulation operations (on items displayed on the screen, e.g., sorting of photographic images or other documents) to widget setting operations without breaking off the task operation flow.

The exemplary widget16thus serves as a menu for its different functionalities (options), which eliminates the need for a conventional menu opening/closing mode, and enables the menu display itself to be dispensed with, by directly associating each virtual side18,20, etc. of the widget with a specific widget setting.

In one embodiment, illustrated inFIG. 2, the widget16has a first functionality associated with a first virtual side18of the widget and a second functionality associated with a second virtual side20of the widget. When the first side18is ON (its functionality is active), only the first side is visible, while the second side20is hidden from view and OFF (its functionality is inactive). When the second side20is ON, only the second side is visible, while the first side18is hidden from view and OFF.

The reversible widget16shown inFIG. 2has only two virtual sides18,20. The association between a side and a widget setting is then two-fold as it merges the setting selection-action with the display-layout of the widget setting itself. In other words, the widget always displays its current setting, and that display at the same time induces the setting. If the display changes (through rotation on itself) the setting changes as well.

In other embodiments, the widget has more than two virtual sides, such as three, four, five, or more virtual sides. Rather than a 2-choice menu implemented through a 2-sided widget, the widget can then be used to provide an N-choice menu, involving, in this case an N-side widget, as exemplified inFIGS. 3 and 4. In one embodiment, a vector of N states (or options) is associated with the widget16and managed as a loop, where N can be any predetermined finite number. The first rotation takes the user to the first widget option; the second one sets the widget to the second option, and so on until option N; and then the N+1 rotation brings the user back to the first possible setting. At any time, in the loop, a new rotation of the widget takes the user to the next option, or the 1stone in the vector if the widget was on the Nth setting. For example,FIG. 3illustrates a widget16with four sides: side1, shown at18, side2shown at20, side3shown at26, and side4shown at28. Only one of these views is visible to the user at any given time.FIG. 4illustrates another widget16with four sides: side1, shown at18, side2shown at20, side3shown at26, and side4shown at28. In this embodiment, the widget16is displayed in perspective view, as a three dimensional object, but only the side which is uppermost is operational at any given time.

As for the widget ofFIG. 2, at any given time, only one of the N sides18,20,26,28of the widget (i.e., those sides that are capable of having an associated operational functionality), is displayed to the user on the user interface, and all the other sides having an associated (inoperative) functionality are hidden from view (invisible or at least partly occluded). In some embodiments, every virtual side18,20,26,28of the widget is associated with a respective functionality.

The widget16can have any suitable shape, irrespective of the number of virtual sides18,20, etc. that it has. Example shapes include a circle, triangle, square, disk, or other multi-sided polygon, polyhedron, or three dimensional object, or even an irregular shape. Thus, a widget16which is circular in shape can have two, three, four, or more virtual sides. In some embodiments, the widget assumes a shape, e.g., with a number of edges in its perimeter, that indicates how many virtual sides it has. For example, the widget may be a polygon with six edges (hexagon) in the case of a six-sided widget. In practice, the number of virtual sides that the widget may have may be up to a maximum of about 10.

The widget16is translated (flipped) between its two (or more) sides by a natural touch gesture on the touch-screen. In flipping a circular disk shown inFIG. 2, for example, the height of the widget16along its vertical axis30remains constant while the width of the first displayed side decreases to a point on a thin vertical line and the width of the second side then increases to its full width. As an example, the widget16is rotatable about its axis30from the first side18plan view to an intermediate position32, where side1is still partially visible but not operational, and then to a second intermediate position33, where neither side is visible, just the widget's perimeter, then to a third intermediate position34, where side2is partially visible but not operational, and finally to the side2plan view at20. From the second position20, continued rotation in the same direction returns the widget to the first position18. In some embodiments, rotation in a reverse direction also brings the widget back to the first position. This is useful in the widget ofFIG. 3, for example, where the widget has more than 2 sides, since the rotation direction can be selected to bring the widget to the desired side with the fewest number of gestures.

In one embodiment, the flipping action may be graphically shown to provide visual feedback of the action. For example, the widget assumes a sequence of intermediate positions between the first and second (or more) sides, e.g., as an animation, similar to a video of a coin being flipped over. As will be appreciated, the three intermediate positions shown inFIG. 2are only exemplary of the possible intermediate positions. In other embodiments, the widget16may be displayed as flipping directly between sides without visualizing any intermediate positions.

To choose between the two (or more) settings of the widget, the user operates a rotation (flipping) of the widget graphic object on the screen to make it revolve on its axis30parallel to the screen plane, virtually rotating the widget through 180°. In this movement, the widget does not move in a 2D space, i.e., there is no change in coordinates of the widget on the screen. Rather, there is only a change of the appearance of the widget at the same location. The widget then exposes the new side that was not visible before the rotation. The new side visible after the rotation completed displays a widget option36that indicates the current widget functionality setting. A subsequent rotation triggered by the user on the widget causes the widget to flip again and the widget to switch to the first or another option38, which is displayed. In the embodiment ofFIG. 2, the two sides18,20are opposed, as in a coin, and the flipping of the widget graphically illustrates the flipping of a coin. The same graphics can also be used for a multi-sided (more than two-sided) widget.

The entire perimeter39of the exemplary widget16has the same functionality which depends on the side displayed, i.e., there is no “north” and “south” region of the perimeter (as in magnets disclosed in application Ser. No. 13/031,336 and Pub. No. 20100313124), although such an embodiment is also contemplated. Thus a graphic object22,24can respond to the functionality of the widget16irrespective of where it is angularly positioned, be it above, below, left, or right of the widget on the screen.

Various touch gestures are contemplated for effecting the translation (“flipping”) of the widget between sides.FIGS. 5 to 9show example gestures, each using one or more of the fingers40,42of one of the user's hands44. The gesture shown inFIG. 5is referred to as a pinch-in gesture, indicate by arrows A. The pinch-in gesture is initiated by bringing a thumb40and index finger42into contact with a target element (the widget16in this case) a small distance apart, then bringing the thumb and index finger together, thumb and index finger then simultaneously lifting off the screen immediately thereafter.

The gesture shown inFIG. 6is referred to as a pinch-out gesture, indicated by arrows B, and is essentially the reverse of the pinch-in gesture. The thumb40and index finger42both come into contact with the widget at the same point on the screen, then move apart, and then are lifted simultaneously away from the screen.

The gesture shown inFIG. 7is referred to as a flick gesture, indicated by arrow C. This gesture can be a flick-left gesture, such as a quick movement of a user's finger that makes initial contact with the screen, (on the widget), brushes the finger in a specific direction (left in this case), and then lifts off the screen. A flick-right gesture can be the reverse of that shown.

In the embodiment shown inFIG. 8, a user applies a gesture on the widget16, indicated by arrow D, whose angle α and/or direction, with respect to a reference axis46, determines to which side the widget is to be rotated. The reference axis46of the widget may be defined as being perpendicular to the rotation axis30, for convenience. As shown inFIG. 9, each of a plurality of sides can be associated with a different angle α and direction. A flick at the specific angle can thus be associated with a specific widget side and a corresponding functionality. For example, the four sided widget shown inFIG. 9is associated with four gestures, one for each of the virtual sides. Rotating the widget through a flip gesture at angle αYflips the widget directly to side Y and hence sets the widget with option Y, without the need for displaying all the sides in between. In the embodiments ofFIGS. 8 and 9, the vertical axis is maintained as the pivot axis30for the rotation, but the angle α of the flicking gesture is recorded with respect to the horizontal axis46.

InFIG. 9, each of the flicking actions begins in the same semicircle of the widget16in order to allow a greater control of gesture detection and a more comfortable operation for the user. Additionally, the number of gestures may be limited to about four of the easiest angles to draw. These gestures are each associated with one of the four virtual sides of a widget16for modeling 4-choice menus. As examples, the four gestures shown are:1: vertical flick top-down,2: horizontal flick right-left,3: 45° angle, flick right-left,4: 45° angle, flick left-right.

Gesture 1 rotates the widget to side1displaying option1; gesture 2 rotates the widget to side2displaying option2, and so forth.

An advantage of such a system is that it avoids having to go through the m options of a menu before being able to reach the m+1 desired option. The appropriate gesture takes the widget directly to the desired setting out of the four possible ones of this widget. Thus, for example, a single gesture at angle α3, when the widget is displaying side1, flips the widget directly to side3.

Other gestures are also contemplated, such as a single tap, briefly touching the surface of the screen with the fingertip over the widget, which triggers one rotation. For an N-sided widget, the number of taps may designate the side to be displayed: 1 tap for side1, two taps for side2, and so forth . . . . A single tap may also be used for rotating the magnet from side (option) m to side (option) m+1. Alternatively, to avoid the need for multiple taps, a “flashing” or “folding” mode may be employed, where the first user's tap or flick on the widget triggers a rotation, which does not stop on the next widget side but continues to slowly rotate through the possible sides, displaying one side after the other, until the user operates a second contact, such as a tap on the screen, to stop the widget rotation on the desired side and option.

It may be noted that, in general, the pinch gesture requires a sufficient space, and generally a sufficiently large widget surface, to be comfortably operated by the user and accurately detected by the TUI. Therefore, it may not be the best suited gesture for small screens, such as smartphones screens. There, the single tap (or plural tap) may be a more accurately recognized gesture to trigger the rotation. The flick gesture generally can fit on both large and small screens. In addition, it offers extra possibilities in that many devices allow not only recognizing the flicking gesture with initial and final contact point coordinates but also capturing the angle at which the flick is operated, as in the embodiments ofFIGS. 8 and 9.

The widget flipping action is thus associated with a specific gesture that is both natural for the user and accurately recognizable by the system.

In the illustrated embodiments, only one-hand gestures are used to operate the widget rotation. However, it is to be appreciated that the design could also allow users to exploit both hands to operate the widget settings.

Once the user is satisfied that the currently displayed side of the widget is the one the user wishes to select, a confirming gesture, such as one or two taps on the center of the widget, may be employed to activate the widget with that functionality. Alternatively, the widget may adopt the functionality of the currently displayed side without further confirmation, once the rotation is complete.

As illustrated inFIG. 1, the exemplary TUI1includes a computer system50communicatively connected with the display device10. The computer system50includes main memory52which stores instructions54for performing the method disclosed herein and a processor56, in communication with the main memory52, for executing the instructions, as well as controlling other operations of the TUI1.

Data memory58stores information60relating to a set62of the graphic objects22,24, etc., displayed on the screen. In one embodiment, each displayed graphic object22,24, represents a respective item, such as a document68,70stored in local memory58or accessible in remote memory. A document, as used herein, can be a text document, an image (e.g., a graphical or photographic image), a combination thereof, or the like. The information60may include the digital documents68,70themselves, and/or information extracted therefrom, referred to herein as document attributes72. Data memory58may be resident in the computer system50or remote from the apparatus1and communicatively connected therewith. Each graphic object22,24, etc. comprises an arrangement of pixels, such as an icon, a thumbnail image of the respective document68,70, or the like.

The computer system50includes one or more input/output devices74,76, for communicating with external devices. For example, digital documents68,70are received via an input/output device74. The display device10is linked to a second input/output device76by a link78, such as a wired or wireless link, for communicating signals from the touch-screen14to the computer system50and for receiving signals from the computer system which cause the display12to change the visualization. Components of the computer system50may communicate via a data control bus80. While not required, the apparatus1may further include one or more additional user input devices, such as a keyboard or keypad82, a cursor control device84, or the like, integral with or separate from the display device.

The computer system50may be a dedicated computing device which is mounted to or otherwise proximate to the display device, or may be PC, such as a desktop, a laptop, palmtop computer, portable digital assistant (PDA), server computer, cellular telephone, tablet computer, pager, combination thereof, or other computing device(s) capable of executing instructions for performing the exemplary method.

The memory52,58may be separate or combined and may represent any type of non-transitory computer readable medium such as random access memory (RAM), read only memory (ROM), magnetic disk or tape, optical disk, flash memory, or holographic memory. In one embodiment, the memory52,58comprises a combination of random access memory and read only memory. In some embodiments, the processor56and memory52and/or58may be combined in a single chip. The interface74and/or76allows the computer to communicate with other devices via a computer network, such as a local area network (LAN) or wide area network (WAN), or the internet, and may comprise a modulator/demodulator (MODEM), USB port, or the like. Memory52stores instructions for performing the exemplary method shown inFIG. 10as well as the processed data.

The digital processor56can be variously embodied, such as by a single-core processor, a dual-core processor (or more generally by a multiple-core processor), a digital processor and cooperating math coprocessor, a digital controller, or the like. The digital processor56, in addition to controlling the operation of the computer system50, executes instructions stored in memory52for performing the method outlined inFIG. 10.

The exemplary display device10can include an LCD or plasma screen, computer monitor, smartphone screen, or the like, which may be capable of displaying in color. The touch-screen14is integral with or overlaid over the display screen12and includes multiple actuable areas which are independently responsive to touch or close proximity of an object (touch-sensitive). The actuable areas may be pressure sensitive, heat sensitive, and/or motion sensitive. The actuable areas may form an array or invisible grid of beams across the touch-screen14such that touch contact within different areas of the screen may be associated with different operations.

Exemplary touch-sensitive screen devices10which allow finger-touch interaction, which may be used herein, include the Multi-Touch G2-Touch-screen or G3-Touch Screen from PQ Labs, California (see http://multi-touch-screen.net), an infra-red grid system, such as the iTable from PQ Labs, a camera-based system, such as Microsoft Surface™ touch-screen table (http://www.microsoft.com/surface/). On this type of large touch-screen, which may have a height and/or width greater than the maximum dimension of an item (e.g., a screen measuring about 80 cm or greater on the diagonal), a large number of digital objects can be displayed and manipulated by one or more users through natural gestures. However, it is also contemplated that the display device may have a smaller screen, e.g., the size of a mobile phone screen, which may display fewer than all the graphic objects at one time (or all of them in reduced size), and provide for viewing a selected portion of one of the documents on the screen. As will be appreciated, where the finger or implement is detected by a camera rather than through pressure, “detecting a touch contact,” and similar language, implies detecting a finger or other implement on or near to the screen, which need not necessarily be in physical contact with the screen.

A variety of different functionalities can be associated with the widget16. In the embodiment shown inFIG. 1, for example, each graphic object22,24represents an underlying digital document68,70, etc., stored in memory58. Attributes72extracted from each document68,70cause the graphic object22,24representing the respective document to respond to the widget in accordance with the current functionality of the widget, e.g., by moving relative to the widget.

The attributes72may be related to textual content of the document (e.g., presence/absence of certain keyword(s), co-occurrence of certain keyword(s), frequency of occurrence of certain keywords, etc.); image content of the document (e.g., presence/absence of images, semantic content of images present, etc.); document type (e.g., email, Word document, spreadsheet, PowerPoint presentation, etc.); document size (e.g., small, medium, large); date of the document (e.g., before a certain date/after a certain date, within a certain data range); or a combination thereof.

In some embodiments, the documents68,70are text documents and may each comprise a body of searchable text. The documents, in some cases, may be a subpart of a larger document and/or which may also include other information, such as drawings, photographs, and the like. In general, a text document comprises one or more text strings expressed in a natural language, such as English or French, and having a vocabulary and a grammar. Each text string of the document can be as short as a phrase or clause of a sentence and generally comprises a sentence and may comprise two or more contiguous sentences. A text document generally includes a plurality of such text strings. Each text string may include a plurality of text elements, such as words, phrases, numbers, and dates, or combinations thereof. Exemplary text documents include word documents, spreadsheets, PDF files, scanned documents which have been OCR processed to extract text, web pages, XML files, and the like. In the case of input XML documents and web pages, the searchable text strings may include hidden text.

For example, side1of the widget16may be associated, in memory56, with a functionality which is selective for documents68which are responsive to a first query, e.g., which include a first keyword. Side2may be associated, in memory, with a functionality which is selective for documents70which are responsive to a second query, e.g., which include a second keyword, or which are not responsive to the first query (i.e., which do not include the first keyword). When the first side18is displayed, graphic objects22representing documents68with an attribute72that is responsive to the first query (e.g., the document includes one or more words corresponding to the first keyword), respond to the widget. By “respond,” it is meant that the graphic objects, or at least some of them, move across the screen, e.g., towards or away from the widget. The keyword search may be applied to the displayable text of the document, to metadata of the document, or to a combination thereof.

The actuated widget16can be moved across the screen12using a finger touch to position the widget close to a specific group of graphic objects22,24displayed on the screen that are of interest to the user. In reaction to the widget movement/position on the screen, some graphic objects may be attracted around the widget. For example, graphic objects22having the attribute72corresponding the actuated side of the widget respond by moving towards the widget16, while graphic objects24representing documents70without the attribute72corresponding to the query respond in a different way, for example, remaining stationary or moving away from the widget. The attracted objects22may correspond to documents evaluated as “relevant to a certain category”, or they can represent documents containing some specific keywords, and so on, depending on the functions and algorithms associated with the widget or widgets.

The collection62of graphic objects can be initially displayed on the touch-screen in various icon arrangements, such as tiles, clouds, icons, or lines, etc.FIG. 11illustrates a cloud arrangement. The widget16may have a home position90in which it is inactive, but remains visible on the screen. When moved off the home position, the widget16may automatically assume the functionality of the displayed side18. Alternatively or additionally, a widget16may be called into view by pressing a button92on the screen, which causes the widget to be displayed. Once a functionality has been selected, the activated widget16can be moved close to a set of graphic objects, where it can attract or otherwise cause a responsive movement by some of graphic objects (which correspond to the documents meeting the selected functionality of the widget).

The different retrieval functions that the widget16can be associated with may include one or more of:

1) Document filtering, e.g., any rule that enables documents to be filtered out, for example, through predefined keyword-based searching rules;

2) Document similarity, e.g., any algorithm capable of identifying a degree of similarity between a predefined sample document and any other document, based on its content;

3) Document classification, e.g., any automatic classifier implemented through an algorithm which is able to associate a predefined label to a document, based on its textual and/or visual content;

4) Document clustering, e.g., any algorithm which groups documents, based on similarity of their attributes (e.g., textual content), into a predetermined number of groups.

As an example, a user may use the widget16for forming clusters of displayed graphic objects, for example, by positioning the widget16near the set62of graphic objects. Those graphic objects whose underlying documents have an attribute responsive to the query group around the widget to form a first cluster, while the remaining graphic objects (or at least some of them) form a second cluster by grouping themselves together further from the widget or by remaining in place. The user can touch a selected one of the graphic objects which causes the underlying document68,70to be displayed on the screen. A clustering application is discussed in further detail below.

In the embodiments illustrated inFIGS. 2-4, the sides are denoted side1, side2, etc., but in practice, any textual representation36,38of the functionality of the side, and/or other representation of the widget functionality, may be displayed. For example, if side1corresponds to a keyword query, the keyword may be displayed on that side. Or, if one side denotes “relevant”, based on some function of attributes72of the documents, the side may display the word “relevant.” The other side could display the word “non-relevant.” Other ways of distinguishing the sides from each other and/or for representing their functionalities are also contemplated, such as through color, texture, display of an image, or the like.

In some embodiments, the widget16is applied to a single document68,70that is displayed on the screen, rather than to a group of graphic objects22,24, etc. representing a collection of documents. In this embodiment, the widget is selective for instances within the displayed document68,70that match the current functionality of the widget, e.g., which are responsive to a query, such a keyword. For example, one side18of the widget is associated with a functionality which is selective for instances of a first keyword (e.g., “Joe” in theFIG. 11illustration) when the document is open. This side18may also be selective for graphic objects representing documents which include the first keyword when only the graphic objects are displayed. The instances94of the keyword may be highlighted or otherwise identified in the open document68. The exemplary widget16thus finds application for document categorization and retrieval as well as for information extraction.

Two or more of the multi-sided widgets16can be employed to provide different functionality combinations. For example, as shown inFIG. 1, two widgets16,100may displayed on the screen12, each one having a choice of functionalities.

With continued reference toFIG. 1, to enable users to interact easily with document sets, the computer system50includes various software components10,112,114,116,118,120,122stored in memory52, which are implemented by the processor56. These may include a widget functionality component110, which implements algorithms and retrieval rules that may be preconfigured by the system and/or the user, for implementing the predefined user's gestures on the widget, described above. The widget functionality component110receives signals from a touch recognition system112and supplies control signals to a display controller114for controlling the widget16, graphic objects22,24, and related functions in response to the touch signals. The configuration of the widget(s)16,100may be achieved through prior user input uploaded by the system at the beginning of the application session, such as by parsing a configuration file or by keyboard input. The configuration setting of a widget may either be fully static (e.g., a fixed filtering threshold can be recorded into a widget whose role is to measure the similarity with a specific document or image); or it may be made dynamic through conventional menu selections.

The graphics for the widget flipping action can be implemented, for example, using suitable software, such as the PyMT: Python Multi-touch library (see: http://pymt.eu/docs/api/api-pymt.ui.widgets.flippable.html). PyMT is an open source library released under the terms of the GNU LGPL, for developing cross-platform multi-touch applications (Linux/OSX/Win).

The touch recognition component112receives electrical signals and/or software information comprising touch coordinates from the touch-screen14of the display device10in response to user touches. In particular, user touches on the screen (or proximate to it) are converted by the display device to electrical signals which are interpreted by the touch recognition component112as corresponding to one of the predefined gestures (or not, if there is insufficient similarity between the detected touches and the predefined gesture). The touch recognition component112may include or access a gesture recognition library which stores gestures and their corresponding meaning.

The display controller114controls the rendering of the widget16and controls the other content of the display, such as graphic objects22,24, and documents68,70on the display, based on the detected gestures and other information. The touch-screen device may come preloaded with a component that provides the touch coordinates of every contact point on the screen, and/or a library with an API that is capable of recognizing some basic and common gestures from a set of touch coordinates.

A retrieval component116is configured for retrieving an underlying document68,70, corresponding to a displayed graphic object22,24, which a user wishes to open on the screen12.

The system50may include one or more document processing components118,120,122for extracting document attributes72and/or to determine whether a document68,70is responsive to a query. Some of these components may function as described, for example, in application Ser. No. 13/031,336 and/or Pub. No. 20100313124. The document processing components118,120,122may include, for example, one or more of a natural language processing component118, a classifier120, and a language identifier122. As will be appreciated, the document processing components can be called remotely from a distant server receiving requests from the system50after user gesture recognition and do not need to be stored local to the TUI.

The natural language processing component118may include a parser for parsing text of the documents68,70to identify words and their morphological properties (noun, verb, etc.), for named entity recognition, and optionally for extracting syntactic relations between words or phrases in a sentence (e.g., between a subject and its main verb, between a verb and its object, etc.), by applying a set of grammar rules, as described for example in U.S. Pat. No. 7,058,567, entitled NATURAL LANGUAGE PARSER, by Salah Aït-Mokhtar, et al., the disclosure of which is incorporated herein by reference. The parsed text may be used for querying or as input for other functionalities, such a language guessing, classifying, and document clustering.

The classifier120may have been trained to assign a document to one of a set of two or more classes, based on its content e.g., based on the occurrence or frequency of occurrence of each of a set of one, two, or more words. Such a classifier may have been trained with a set of training documents that are each labeled with a category label selected from a finite set of labels corresponding to the categories.

The language identifier (LI)122may include a set of rules for language identification (a language guesser) as disclosed, for example, in G. Grefenstette, “Comparing Two Language Identification Schemes”, in Proc. 3rd Intern'l Conf. on the Statistical Analysis of Textual Data, JADT'95, December 1995, Rome, Italy; U.S. Pat. No. 5,062,143, issued Oct. 29, 1991, entitled TRIGRAM-BASED METHOD OF LANGUAGE IDENTIFICATION, by John C. Schmitt; and Jean-Pierre Chanod and Pasi Tapanainen, “Tagging French text: tagset, dedicated lexicons and guesser,” Proc. From Texts To Tags: Issues In Multilingual Language Analysis, EACL SIGDAT workshop, Dublin (1995), the disclosures of which are incorporated herein by reference. In an exemplary embodiment, the language guesser includes rules designed to identify at least two languages and the widget functionality is such that a first side of the widget is selective for documents which are predicted to be in a first of the languages and a second side of the widget is selective for documents which are predicted to be in a second of the languages. For example, when the first side18is active, English language documents (those with at least a threshold probability of being in English, according to the LI component) may be separated out from the other documents while when a different side20is displayed, French language documents are separated out.

The initial setting of the widget determines which of these document processing components118,120,122is used to implement the widget functionalities that are available to the user by flipping the widget. Multiple widgets may be provided which have different ones of these functionalities.

As will be appreciated,FIG. 1is a high level functional block diagram of only a portion of the components which are incorporated into a tactile user interface1. Since the configuration and operation of programmable computers are well known, they will not be described further.

FIG. 10briefly illustrates an exemplary method for using the widget16. The method starts at S100. It is assumed that a user interface1is provided with appropriate software54stored in memory.

At S102, when a user wishes to make use of the widget functionality, the widget application software110is loaded and a widget16is automatically displayed on the screen, or can be called up by the user using the widget button92. By way of example, the welcome screen of the application provides a number of magnets that are displayed on the screen for different applications, such as keyword filtering, clustering, and so forth. An indication of the type of functionality they can each provide, e.g., a short textual description, may be displayed on top of each inactive magnet. In some embodiments, the user may use a conventional menu for selecting functionalities for populating each of the sides18,20, etc., if this has not already been set. When activated (or also when inactive) the displayed virtual side18shows a representation36of its functionality.

With a widget16displayed on the screen, a predetermined touch gesture (such as one of those shown inFIGS. 4-9) on the widget location (e.g., one or more finger movements directly on top of the displayed widget16), corresponding, in memory, to a request for a change in the functionality of the widget, is detected by the system (S104).

At S106, the corresponding transformation of the widget between sides18,20is graphically displayed, e.g., as an animation, and a representation38of the functionality associated with the selected virtual side20is displayed at the end of the transformation. The system50may wait for a confirming gesture by the user to actuate the widget16with the functionality of the displayed side or may do so automatically. The current side setting is stored in memory.

At S108, a predetermined touch gesture corresponding to the movement of the widget is detected by the touch recognition component112and a graphic animation of the moving widget is generated by the widget functionality110and displayed on the display device by the display controller114.

At S110, when the widget16is active to elicit a response in the graphic objects, e.g., positioned by finger touch within a predetermined distance of the graphic objects22,24or otherwise positioned closer to the graphic objects, this causes the graphic objects to exhibit a response. The response of each of the graphic objects to the current functionality of the widget is computed. This may be performed by the widget functionality110, optionally in cooperation with one or more of the document processing component(s)118,120,122and/or based on the stored attributes72. The corresponding response, e.g., movement, relative to the widget, of some or all of the graphic objects, is displayed. Since the exemplary widget functionality is the same around the entire perimeter39of the widget16, the widget elicits the same response from a graphic object (moving towards the widget, moving away from the widget, moving to a cluster, or staying put), irrespective of the object's initial angular position, relative to the widget, on the display screen12. In some embodiments, the widget may have a predefined action perimeter, which limits the graphic objects on which elicits a response to those within a predefined radius or other predefined area.

At any time, the user may use the widget setting gesture to select a different side of the widget (a return to S104), and the surrounding graphic objects respond accordingly.

At S112, an item68,70corresponding to one of the displayed graphic objects may be called up onto the display by a user, for example, with a touch gesture on the corresponding graphic object22,24. The widget16can then act on the displayed document itself, based on the functionality of its current side.

The method ends at S114.

The exemplary method may be implemented on one or more general purpose computers, special purpose computer(s), a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, a hardwired electronic or logic circuit such as a discrete element circuit, a programmable logic device such as a PLD, PLA, FPGA, Graphical card CPU (GPU), or PAL, or the like. In general, any device, capable of implementing a finite state machine that is in turn capable of implementing the flowchart shown inFIG. 10, can be used to implement the method for using a widget.

Example uses of the widget16will now be described.

In one embodiment, the 2-sided widget serves as a binary categorizer: the widget itself is pre-configured to operate the probabilistic classifier120which can segregate documents into two distinct categories. The user can move the widget16toward a set of objects in order to pull out, from the set, the objects representing documents from one particular category or from the other. Each side of the widget is associated with a respective category, which can be represented, for example, through two colors, such as red and blue.

Rotating the widget to cause the blue face to be displayed will set the widget with a capability of attracting all documents having a “high” probability (according to a predetermined threshold) of belonging to the blue category. Further flipping the widget will immediately result in a widget capable of attracting all documents having a “high” probability (according to a predetermined threshold) of belonging to the red category.

This ability is fully contextual and does not break off the interaction flow in a disturbing way: the user does not need to move or change position to reach a different place on the screen for setting the magnet widget on the chosen or desired functionality. If the widget is currently over a specific set of objects, it can be flipped at its current position to switch from one functionality to another, without further interrupting the user task operation flow, without changing the widget size (e.g., to open a functionality selection menu) or hiding the data underneath through a menu display. In addition, the objects' reaction to a category switch can be made immediately visible to the user (for instance new objects are attracted) without further moves or actions on the widget.

This property applies to the binary classifier widget as well as to any other reversible widget in general. The same functionality can be extended to an N-choice menu, involving in this case an N-sided widget.

In this embodiment, the widget is associated with a classifier120modeling N categories, and a classification algorithm. Each side of the widget corresponds to a category out of the N categories supported by the classifier model and algorithm. The classifier120can include a set of binary classifiers, one for each virtual side (displayed class vs. rest). A widget currently set on its “Category1” side and moved towards a set of objects (documents) will attract all documents having a high probability of belonging to Category1(for example, according to a certain threshold). In other embodiments, a single multi-class classifier, modeling all the N categories together, may be employed. Then, if a category m is visible on the widget top side, the classification algorithm is launched on all target documents corresponding to graphic objects within the widget's action perimeter. All these documents are classified by the classifier against all the N modeled categories. Then, the document scores against the category m alone are considered for each document. Specifically, the value is compared to a threshold (that may be specific to the category m within the classifier mode), and if the probability score meets, e.g., is at or above the threshold, the corresponding graphic object exhibits a response to the widget set on category m. Flipping the widget placed over the same collection of objects (documents), up to another side/category will help pulling out objects (documents) from another category, and so on. In yet another embodiment, the side/categories may simply represent keywords and the classification algorithm120is simply a string matching algorithm, detecting documents holding some particular keywords.

Language Identifier

In this embodiment, the widget16is associated with a language detection/identification system (LI)122, such as an algorithm stored in memory54. Each virtual side18,20, etc. of the widget corresponds to a specific language within the set of languages that are supported for detection by the LI system. A widget currently set on its “Spanish” side and moved towards a set of graphic objects will attract all of the graphic objects representing documents having a high probability of being in Spanish (for example, according to a certain predetermined threshold). Flipping the widget placed over the same collection of documents, to associate the widget with another language will help pulling out objects representing documents in another language, and so on.

Tuning Widget Operating Mode:

In Pub. No. 20100313124, the magnet widgets, by default, attract documents matching the search or retrieval criteria implemented by the widget. The widget's operation mode could be the same or opposite, repelling the documents matching the criteria. To provide the user with the choice, the widget behavior may be tunable to select one of these response criteria as its functionality. This can be implemented through a 2-side widget, where one virtual side of the widget sets the widget in its attracting mode; the other side sets the widget in the repelling mode. The user can flip to one or the other at any time.

Case Sensitivity for Keyword Search

In this embodiment, the widget16is associated with a keyword search algorithm, (see for example, Pub. No. 20100313124). The widget serves to retrieve documents holding a particular keyword. The selection of the particular keyword can be statically pre-configured through the parsing of a system configuration file, or it can be dynamically set in the course of the task operation flow through an on-screen selection (e.g., as described in application Ser. No. 13/031,336).

A more advanced tuning of the search algorithm may allow control of the case sensitivity of the search. This may be implemented through a 2-sided widget16: one virtual side18of the widget makes the query case-sensitive and the other20makes it non-case-sensitive. When the widget is placed over a collection of graphic objects displayed on the touch-screen, flipping the widget over that set helps the user visualize the volume of items matching the query in a case or non-case sensitive mode. If the user initially choses the mode which is case-sensitive, but the search does not retrieve any objects, the extension of the search to a non-case sensitive matching is immediately available to the user.

Clustering Widget

With reference toFIGS. 12 and 13, the widget16may serve as a clustering tool in one embodiment. Clustering in general is the task of discovering homogeneous groups of objects within a data set. The document clustering plays a major role in the range of technologies that can help supporting researchers' activities (e.g., document review, tagging or discovery), especially when dealing with large volumes of documents. Although it is a well-known data mining problem and a number of algorithms have been proposed to solve it, the translation of this functionality onto a multi-touch interface is not straightforward, especially when targeting use of the system by non-expert users.

The clustering widget16or “splitter” has a number of sides, such as from 2-6 virtual sides, each side corresponding to a predetermined (maximum) number of clusters. Each side thus corresponds to a number n of clusters to compute, in the range of 1 to N. Flipping the splitter widget allows the user to select the desired number of clusters to compute. Moving the widget onto a group of documents has the effect of breaking the set of documents into n sets of documents placed around the widget.

The user selects the desired number of clusters by flipping the widget to display the side which displays a representation of that number of clusters. As an example, the side for three clusters may show the number “3”, or three dots, or the like. As will be appreciated, the cluster numbers can proceed sequentially, such as 2, 3, 4, 5, 6, etc. with the sides. In other embodiments, suitable cluster numbers for a given application may be provided, such as 3, 5, 8, 10, which are not sequential.

The exemplary displayed graphic objects22,24, in this embodiment, may represent a set of electronic text documents68,70stored in memory58. Attributes72of the text documents, which cause the graphic object to respond to a widget, in this case, can be based on the presence or absence of textual content, such as words or text strings, or the document content as a whole, such as the frequencies of keywords found in the document, cluster based attributes generated by automatically assigning the documents to one of a predetermined clusters based on similarity, or any other attribute which can be extracted, from the document or from an associated external metadata file, such as date sent, author, metadata, such as document size, document type, image content, and the like. Clustering of documents based on similarity is described for example, in U.S. Pub. Nos. 2007/0143101 (now U.S. Pat. No. 7,813,919), 2007/0239745 (now U.S. Pat. No. 7,720,848), 2008/0249999 (now U.S. Pat. No. 7,711,747), and U.S. Pub No. 2010/0088073, the disclosures of which are incorporated herein in their entireties by reference.

The clustering widget16operates in a mode similar to a magnet widget in its repelling interaction mode. This “splitter” is an N-side widget that operates the clustering of the set of documents on which it has been placed or moved. As an example,FIG. 12shows a set62of graphic objects documents gathered in a cloud. When the splitter16with its functionality set to 3 (clusters) is brought near to the cloud, three non-overlapping clusters130,132,134form, i.e., three groups of graphic objects, each of which is discretely located at a distance from the other groups, as shown inFIG. 13. The clusters may have the same size (same number of graphic objects) or may be of different sizes, depending on the clustering algorithm used and the document attributes considered. Some of the clusters may be empty, i.e., without any documents, and thus the number of clusters visible on the display may be less, in some cases, than the number of clusters selected for the splitter.

Moving the widget16away from the graphic objects cancels the partition and returns the graphic objects to their original grouping. A flick on the widget changes the number of clusters to compute through the splitter.

A suitable touch gesture, such as double tap onto the splitter widget16, makes the partition of the clustering permanent. If the partition has been made permanent, the user can move the splitter widget16towards one of the newly created groups to re-apply the clustering and refine the document partition. As an example, the user may thus make three new sub-clusters from one of the clusters130,132,134, or flip the splitter to create a different number of sub-clusters.

Keywords can be computed by the clustering and returned as a description of each cluster. Each cluster is thus assigned a set of keywords. Some of the keyword sets may be partially overlapping. In general, for any two clusters130,132, fewer than all their keywords are in common, i.e., at least one of the keywords is different.

The exemplary TUI system and method described herein merges the setting selection-action with the display-layout of the widget setting itself. For example, it uses the number of rotations and/or the angle of rotations of the widget on the display; provides simple control and operation of object clustering on a TUI, and provides a natural and understandable visual feedback of the action.

As will be appreciated, the widget16may also find use as a virtual magnet for use in query generation, according to the method described in application Ser. No. 13/031,336, and for other document tasks, as described in U.S. Pub. No. 20100313124 and U.S. Pub. No. 20100312725. For example, the system50may recognize different touch gestures for the widget functionality setting and for query generation or other tasks.