Dynamic view user interface system and method

A mobile computing device has a display and a user interface UI input device that has a control system for manipulating displayed data. The display and UI can be part of a touchscreen or the UI can be decoupled as an input device that is separate from the display. The UI can be used for scrolling, zooming, and selecting specific data on the visual display. The basic operation of the UI includes a touch circle being formed around a control point where the user touches the touchscreen. The user can swipe on the touchscreen with radial and arcuate gestures which can be interpreted by the UI for zooming, selecting, and inputting data.

BACKGROUND

Touchscreen and touchpad user interfaces (UIs) are used on various types of computing devices such as mobile smartphones, tablet computers, smart watches, and some desktop computers. Today, the majority of common touchscreen interactions fall back to the touchscreen analog of point-and-click, which means that user input is directly coupled to the size of the screen. For example, dragging and dropping an object across the screen requires larger and larger movements as screen size increases and becomes too intricate to perform reliably with a finger as the screen size decreases. The rising popularity of touchscreen interfaces has led to a small handful of special-purpose gestures to become commonly used (such as two-finger scroll, swipes, two-finger taps, pinch to zoom, etc.), but more comprehensive non-point-and-click gestural interfaces have failed to capture consumer attention at least in part due to the lack of a sufficiently general interface that provides interactive visual feedback.

Manipulating a large amount of information can be a problem on a small visual display.

As UIs for software running on computing devices become more complex, the number of UI options can also increase. What is needed is a UI system and method which allows more information to be shown on a small display screen so that a user can quickly enlarge areas of focus and easily make accurate selections, even when a large amount of information is present.

A final problem area is consistency of interaction. Traditional point-and-click style UIs often rely on the user to discover and memorize the locations and symbols for widgets, buttons, menus, etc. across applications. These can vary drastically, which requires a significant amount of manual design and testing. This inherent inconsistency can also inhibit or prevent user muscle memory. A UI that offers a consistent interaction across applications and layouts could save user, developer, and designer time.

SUMMARY OF THE INVENTION

Touchscreen UIs on mobile computing devices can have improved efficiency and accuracy when making UI selections can be made through input gestures. The computing device can have a processor coupled to the UI that can detect a first input point of a gesture and the processor can form a touch circle around the first input point. The user can then swipe across the touch screen as part of the gesture input and the user interface can detect a most recent input point in the gesture. The processor can then determine an input vector based on the first input point and the most recent input point of the gesture.

The computing device can store a listing of data blocks and display a visible portion of the listing of data blocks that is displayed on the visual display while a non-visible portion of the listing of data blocks exist on the list but is not visible on the visual display. The visible portion of the listing of data blocks can be controlled by the UI gesture that are input by the user. The processor can associate the visible portion of the listing with different angular regions of the touch circle of the user interface where each angular region is associated with one of the data blocks in the visible portion of the listing. The processor can interpret the input UI gesture and alter a visual zoom level that is proportional of a length of the input vector and a focus on the data block associated with the angular region of the input vector. The visual display can display the visible portion of the listing in a dynamic visualization based upon the assigned zoom level and the focus on the data block associated with the angular region of the input vector. This feature can allow a user to easily zoom and scroll through large quantities of data blocks. The dynamic visualization converts a normal listing of data blocks from a uniform list where each of the data blocks is equal in size and legibility to a focused display where at least one of the data blocks is enlarged in the visible portion of the listing of data and at least one of the data blocks is compressed or shrunk in the visible portion of the listing of data. The dynamic visualization can indicate the increased focus by animating the visible portion of the listing of data based upon changes in the input vector. In other embodiments, the dynamic visualization indicates focus by altering a color of at least one of the data blocks in the visible portion of the listing of data based upon the assigned focus on the least one of the data blocks associated with the angular region of the input vector.

The user interactions with the UI can be used to control the operations of the computing device. For example, the processor can determine a control point based on the first input point and then parameterize a Möbius transformation using the control point. The processor can then use the Möbius transformation to drive a dynamic visualization that alters the visible portion of the listing of data blocks that is displayed on the visual display. The dynamic visualization can associate the data blocks in the visible portion of the listing in various ways. For example, in some embodiments the dynamic visualization can associate the data blocks in circular spaces or spherical spaces. In some embodiments, the dynamic visualization can associate the data blocks in the visible portion of the listing with pairs of points for the angular regions on a circumference boundary of the unit disc.

The UI can detect a final input point in a gesture. The processor can detect a final input vector based on the first input point of the gesture and a final input point of the gesture. The processor can then associate an action for each of the data blocks in the listing. The processor can then determine a selected listing based on the angular component of the final input vector. If the length of the final input vector is greater than a predetermined length to select the selected listing the processor can perform the action associated with the selected listing. The computing device can store in the memory, a history of the selected listings and associated actions. If the computing device detects a back selection through the UI the processor can undo the action associated with the selected listing and remove the action associated with the selected listing from the memory.

In some embodiments, the UI can be used to select data blocks from an initial listing group with a primary gesture and then select a subgroup listings with a subsequent gesture. The listing of data blocks can be displayed and the UI can be used to input a gesture. The computing device can interpret the completed gesture as the selection of a data block in the listing. Each of the listings can have a plurality of sublistings. When the listing is selected, the UI can display the sublisting data blocks that were associated with the listing and stored in memory. The computing device can then detect a subsequent gesture and determine a subsequent input vector based on the subsequent gesture. A portion of the sublisting of data blocks can be displayed and a non-visible portion of the sublisting of data blocks may not be displayed. The user can use a scroll gesture to scroll through the sublisting to locate the desired sublisting data block. The computing device can associate each of the visible sublistings with different angular regions of the user interface and the processor can assign a sublisting zoom level that is proportional of a length of the input vector and/or a sublisting focus on the data block associated with the angular region of the subsequent input vector. The computing device can then display the visible portion of the listing in the dynamic visualization based upon the sublisting zoom level and/or the sublisting focus on the sublisting data block associated with the angular region of the subsequent input vector. The subsequent gesture can be used to select one of the data blocks from the sublisting.

In an embodiment, the computing device can detect concurrent gestures that are used to manipulate sequential listing of data blocks. The computing device can display a single dynamic visualization with the focus obtained from the multiple gestures and assign a zoom level and/or a focus on the visible portion of the listing controlled by both the gesture and the concurrent gesture. In an embodiment, the single dynamic visualization can include a first dynamic visualization and nested individual dynamic visualization.

In some embodiments, the computing device can display a nondynamic visualization of the visible portion of the listing data blocks before the first input point of the gesture is detected by the user interface. The computing device can then transition from a nondynamic visualization to the dynamic visualization after the first input point of the gesture is detected by the user interface. Once the processor determines that a gesture has been completed, the computing device can return to displaying a nondynamic visualization of the listing data blocks.

In some embodiments, the listing has a visible portion that is displayed by the computing device and a non-visible portion that is accessible but not displayed. The UI can be used to scroll the listing so that the data blocks in the non-visible portion can be moved to the visible portion. If the computing device detects that the focus of the gesture has moved past a last data block in the visible portion of the listing of data blocks, the processor can revise the visible portion of the listing that includes subsequent data blocks and display the subsequent data blocks in the revised visible portion of the listing. Conversely, if the processor determines that the focus of the gesture has moved backwards past a first data block in the visible portion of the listing of data blocks the computing device can revise the visible portion of the listing that includes prior data blocks which are before the first data block. The computing device can then display the prior data blocks in the revised visible portion of the listing.

While the computing device can primarily provide visual UI display functionality, in some embodiments the UI can incorporate tactile or audio feedback. For example, in an embodiment, the computing device can include a vibration device coupled to the user interface. The processor can determine a control point based on the first input point and a selected data block from the listing based on the radial and angular components of the input vector. In response to detected gestures, the UI can response with a vibration feedback. For example, the user interface can be vibrated with the vibration device to indicate that the selected data block has become active. In other embodiments, the computing device can include an audio device and in response to detected gestures, the UI can response with audio feedback. For example, the computing device can emit an audio signal to indicate that the selected data block has become active.

In some embodiments, the UI functionality of the computing device can be configured by the user. The computing device can store a maximum drag distance from the start of a detected gesture. If the computing device detects a drag distance beyond the stored maximum drag distance, the UI scrolling can be limited. The computing device can also store a maximum number of listings that can be associated with an angular range on the UI input device. The computing device can limit the amount of angular motion that is applied to an input vector so that when the input vector is longer than the maximum drag distance, angular motion of the input vector through the angular range results in maximal focus traversing through the maximum number of listings.

DETAILED DESCRIPTION

A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. While the invention is described in conjunction with such embodiment(s), it should be understood that the invention is not limited to any one embodiment. On the contrary, the scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications, and equivalents. For the purpose of example, numerous specific details are set forth in the following description in order to provide a thorough understanding of the present invention. These details are provided for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the present invention is not unnecessarily obscured.

The currently dominant touchscreen mobile Us on mobile computing devices are the result of trying to create a UI that is familiar to desktop users rather than a UI that is best-suited for mobile touchscreen devices. What is needed is a user interface (UI) for a computing device having a touchscreen input that can utilize both non-traditional and non-point-and-click touch controls. The inventive UI can improve efficiency and accuracy when making UI selections and also encourages user finger muscle memory.

While the UI can be implemented on larger screens, the technology's development was inspired by wanting to perform complex tasks on a small screen such as a screen for a smartphone. The inventive UI does not require any exotic or special hardware and works with existing input mechanisms such as touch devices, trackpads, and mice, and it could also be implemented using three-dimensional input such as 3D mice and 3D finger motions/gestures or embedded in simulated three-dimensional dimensional environments such as Virtual Reality (VR) and Augmented Reality (AR).

A fundamental characteristic of the inventive UI is a “dynamic visualization”, which can combine: an n-dimensional control point manipulated by the user in the context of a gesture, a visualization based on the control point, a collection of elements to select from, and angular/radial rules for object selection when the user terminates a gesture. In some embodiments, multiple dynamic visualizations are combined, which allows the user to control a single master visualization using multiple inputs (e.g., two or more fingers).

The inventive system can start with a “static view” and convert this into a dynamic view. The static view is an arbitrary layout in an n-dimensional space. Note that even though it is referred to here as a static view, it can include animations, video, etc. that are not directly associated with user input. The static view can be transformed into a dynamic view using an interpolation method. Some embodiments use a static view which is transformed into a dynamic view when a gesture begins and then transformed back into a static view when a gesture ends. This allows for optimal static viewing as well as dynamic view interaction by the user's fingers.

In some embodiments, angular motion of a user's finger swipe gestures on a UI input touchscreen may be used to scroll through elements of the collection, making some elements visible and hiding others. During the finger swipe gestures, angular motion may be used to scroll through elements of the collection, making some elements visible and hiding others. Scrolling can display a portion of the listed data on the display while other portions of the listed data can be hidden above and below, or left and right of the displayed portion of the listed data. This allows some embodiments of the invention to limit the number of collection elements displayed at a time. Some embodiments can subdivide the listed data into a tree of sublistings in order to limit the number of choices that can be made by a single gesture.

Embodiments of the present invention could be implemented on any platform using any number of 2D or 3D inputs, including desktop computers, laptop computers, touchscreen computer Input/Output (TO) devices, smart watches, wearable computing devices, virtual reality (VR) computing devices, mobile and other types of computing devices. The inventive UI can be used for various computing inputs including but not limited to: keyboards, nested menus, cursor navigation, text region selection, web pages with interactive elements, image viewers, contact database navigation, file browser controls, completion menus, context menus, video game heads up displays (HUD), window management, typing, text editing, software code editing, etc. Note that because the UI decouples user input from its visual representation spatially, the input and output need not reside on the same device. For example, hand gestures could be captured by a camera, or a touch device could be used to control a much larger screen, display wall, etc.

Touch Circles and Touch Spheres Inputs

The mechanics of the invention are computed based on a gesture performed by a finger touch, drag, and release on a touchscreen, a mouse, pointing stick, or joystick click, drag, and release, etc. A useful construct for discussing such gestures is a “touch circle”. A touch, swipe and release of a finger on a UI touchscreen can be one type of gesture. The touchscreen can detect the points of contact, swipes, and releases and the computing device can interpret the detected finger gestures. The input gesture can begin with a touch at an initial control point and the UI can respond by creating a touch circle that can be an imaginary circle placed around the initial touch control point. The radius of the touch circle can be a predefined constant that could be an application setting. Touch circles can be used as a basis for creating control points that are normalized based on the associated touch circle's radius, so that input points can be transformed so that points inside the touch circle land inside the unit disc. This allows for a representation of touch points that is independent of the touch circle radius and effectively establishes a radial scale for gestures.

FIG. 1illustrates an embodiment of a touch circle1001overlayed on the display of a mobile computing device1003. The user can touch a first control point1002and a touch circle1001can be thought of as forming around the first control point1002. The user can then swipe from the first control point1002along a non-linear path1004to the second control point1005where the user may release the touch contact with the touchscreen UI. The computing device1003can also display information on the touchscreen UI and the described interactions can be interpreted by the computing device to manipulate the information displayed. Some manipulations of information can be scrolling and zooming which will be described in more detail. Note that the location of the touch circle is determined by the initial touch point and that there is effectively no limitation on where the user needs to touch the screen. A graphical representation of the touch circle itself may be displayed but is not fundamentally necessary.

In other embodiments, the UI can utilize touch spheres rather than touch circles. Touch spheres are the analogous construct in three dimensions. Touch spheres could, for example, be manipulated using a 3D mouse or by detecting the length and orientation of a vector between two fingers or a finger and thumb in three-dimensional space.

FIG. 2is an embodiment of a touch sphere which is analogous to the touch circle embodiment illustrated inFIG. 1. In this spherical touch point embodiment, a touch sphere1101is illustrated on the display of the computing device. The user can initiate the touch sphere1101at a first control point1102and move a control point outward along a path1103to a control point1104on the outer spherical surface of the touch sphere1101. The UI can manipulate the information displayed by scrolling, zooming, and/or selecting data blocks which will be described in more detail.

When other input UI devices are being used such as pointing sticks and joysticks, these sensors can have a stick that is moved by a user. The tilt of the stick must first be translated into cumulative motion in the normal fashion in order to obtain a location inside a touch circle. The direction of the joystick can determine a movement direction of a cursor and an angle to the joystick can be proportional to velocity of the cursor. The joystick may also have buttons which can be used to initiate a touch. Thus, a user can press a button to place a control point and then the joystick can be manipulated to swipe to another control point. The user can repress or release the button to perform the touch release to complete the gesture input.

In some embodiments, movement outside the touch circle or touch sphere potentially with some margin can be treated as though the movement outside the circle or sphere were truncated at the boundary of the circle or sphere, and in other embodiments every time a control point lands outside the boundary, the center of the circle or sphere is moved such that the new control point is on the boundary in the same direction as before.

Other embodiments may have special rules for touch circles or spheres when the user's finger or cursor is at the edge of a physical input area (e.g., the edge of a touchpad). For example, a center could be established such that it is possible for the user to still have full range of motion for the touch circle, or an animation could be used to move the touch circle's center inward while the finger or cursor remains close to the boundary of the physical input area.

FIG. 3is an example of truncating the touch point when a finger swipe1202moves outside the touch circle1201. In this example, the user has touched the control point1204and then swipes in a path1202that extends outside the touch circle1201and then re-enters the touch circle1201before stopping at control point1205. Rather than attempting to interpret the movement outside the touch circle1201, the UI can create a truncated path1203that runs along the inner edge of the touch circle1201from the exit intersection of the touch circle1201and the swipe path1202to the entry intersection of the touch circle1201and the swipe path1202. This truncation path1203guarantees that the control points1204,1205will remain inside the unit disc touch circle1201. This UI feature can also be applied to touch spheres.

FIG. 4illustrates another embodiment of a UI system for ensuring that touch points translate to control points that remain inside the unit disc. In this embodiment, the user has initially touched the touchscreen at control point1304. The user then swipes along the swipe path1303which contacts the touch circle1301. Rather than allowing the swipe path1303to exit the touch circle1301, the UI can move the touch circle1302with the user's swipe path1303until the second control point1305is reached. This UI feature can also be applied to touch spheres.

FIG. 5illustrates an edge case where a user touches close to the edge of the screen1401of a computing device. Normally, the UI will create a touch circle1402with the first control point1404at the center of the touch circle1402. However, if the first control point1404is too close to the edge of the display, there is insufficient room to swipe right on the screen1401and the functionality of the UI will be compromised. In order to prevent this type of UI malfunction, the Lit can respond to a first control point1404being too close to the edge of the display by moving the touch circle1403away from the edge of the display1401. In this example, the UI can place or move in an animated manner the touch circle to the left of the edge to provide space to use the whole touch circle1403. The UI will move the touch circle to the right if the first control point is too close to the left edge of the screen, move the touch circle down if the first control point is too close to the top edge of the screen, and move the touch circle up if the first control point is too close to the bottom edge of the screen. Similarly, combinations of movements can occur if the first control point is too close to a corner of the display screen1401by moving the touch circle down and left if the first control point is too close to the upper right corner, moving the touch circle down and right if the first control point is too close to the upper left corner, moving the touch circle up and left if the first control point is too close to the lower right corner, or moving the touch circle up and right if the first control point is too close to the lower left corner. This UI feature can also be applied to touch spheres.

Selection and Actions

Special UI control rules may be necessary for determining when a user has made a selection through the UI controls. Specific actions or gestures can be associated with UI user selections, so that making a user selection changes the state of the UI control application in a way that may be configurable by software on the client's computing device. Once a selection has been made, a control point may be stored to represent each user selection that is made, so that a visualization can be frozen for display and/or reversed by a back visualization which can be a graphical animation displayed on the visual display. This stored control point may be the control point at the time of selection, or the stored control point may be some canonical control point representing that selection (e.g., a control point of length one that is angularly centered in a disc sector associated with the selection). If a canonical control is used, an implementation may trigger an animation from the current control point to the canonical one.

In an embodiment, the touch circle can be associated with a “scroll disc.”FIG. 18illustrates an embodiment of a “scroll disc”128used for scrolling control. The user can touch the touchscreen at an initial control point and the touch circle will form. From the initial center control point the user then swipes to a release control point where the user can release the touch control point. If a release happens inside the neutral zone201, the control point moves or animates back to the center. If a release happens inside the back selector205, the back action is triggered and the UI can go back to the prior selection. If the control point enters the scroll selector123, the user can scroll through data and the back selector205becomes inactive for the rest of the gesture until the release. If a user releases while in the scroll margin207then a selection is made. In some embodiments, user selections may fail if they are within some angular margin of error of selection bounds. The user can perform the intended gesture again to perform the desired selection.

In some embodiments, a special back selection control action is possible through a gesture input to the UI. When a back selection is made, an associated back action is triggered. For example, the most recent selection could be reversed and the visual display may revert to a prior selection or screen display so that selection errors can be corrected by repeating the gesture inputs described above.

Because there may not be an outer radial bound on the region in which a gesture termination will successfully trigger a selection, selection accuracy improves with the drag length of the gesture. This is considered to be an improvement over systems where selections are based upon a closed area or volume (e.g., clicking or tapping a button on the UI).

The length of the gesture required to make any selection is bounded by and proportional to the size of the touch circle or touch sphere, so the length gesture required to perform any UI interaction can be minimized. This is considered to be an improvement over systems where operations are based on visual layout (e.g., tapping buttons or selecting from menus via point-and-click places no bound on the amount of motion required).

Gesture Stack

Multi gesture lookahead can be supported by the UI system for maintaining a stack of touches and associated control points. This allows for interleaving user input gestures and their visualizations in a similar fashion to interleaved motions while typing on a physical keyboard. In other words, more than one gesture can be simultaneously input by the user and interpreted and processed by the computing device at the same time. Each gesture can affect its own displayed visualization and hence the overall scene of data manipulation on the visual display. The current and upcoming visualizations can be thought of as a decision tree. If a second gesture begins while an earlier first gesture is still happening, the second gesture can be mapped to the entire next level of the decision tree for its duration. This means that the second gesture effects can potentially be seen on many nested visualizations in parallel on the display. The first gesture is thought of as the primary gesture, and if the first gesture makes a selection successfully, any gesture mapped to the next level of the tree becomes the primary gesture after the first gesture selection is completed. New gestures can be mapped to the first level of the tree that does not have an associated gesture. In some embodiments, non-primary (secondary) gestures cannot select, and in others, non-primary gestures can make selections that are frozen and finish processing if the first gesture selection is successful and the next tree level becomes primary.

Disc and Ball Automorphisms

In some embodiments, the UI visualizations can be achieved using an automorphism on the 2D unit disc or the 3D unit ball or sphere parameterized by, respectively, a 2D or 3D control point. A disc automorphism is a continuous mapping from the interior of the 2D unit disc to itself, and similarly, a ball automorphism is a continuous mapping from the interior of the 3D unit ball to itself. One useful disc automorphism PHI φ can be defined as a Möbius transformation:

Where: c is the control point and both c and z are complex numbers.

PHI φ can have many interesting properties, including but not limited to mapping the boundary of the unit disc to itself, and 2D display mapping circles to circles. Similar Möbius transformations can be used to define an automorphism on the 3D unit sphere.

Input Mechanics

An input mechanic can be defined by a set of rules for transforming user input into data that, in combination with a set of listings and their associated data, can be used to drive visualizations shown on the display of the computing device. These visualizations may not require any extra coupling with user input. For example, a touchscreen visualization may not need to be displayed in a location and size centered on or adjacent to the position of the user's finger(s) on a touchscreen of the computing device. In particular, a visualization must make use of a “primary mechanic” (described below), possibly in some combination with a “back mechanic”, a “grouping mechanic”, a “scroll mechanic”, or a “transition mechanic”. Only the primary mechanic is necessary to form an input mechanic that can drive a visualization, and the other mechanics may be mixed and matched as necessary by the UI system.

Primary Mechanics

A primary mechanic on the inventive UI system is a technique for deriving control points from some region of a touch circle (or the whole touch circle) and using the control points to drive a visualization displayed on the display of the computing device based on a group of user input selections. Usually some interesting aspects of the visualization are tied to the location of the control point, such as the degree that selections are zoomed, their degree of focus/highlighting, etc. For some primary mechanics, the output is the data necessary for a visualization to position and scale a visual representation for each selection. In an embodiment, a visualization of data on the display may temporarily discard or conceal the visual representations for some selections determined to be of negligible interest to the user. In other embodiments, the visual output shown on the display can be a degree of focus that can be applied to each user selection. Still others embodiments can allow the UI visual output to position, scale, and focus information together.

Primary mechanics can be classified based on both the dimensionality of the control points they operate on and the dimensionality of the visualization input data they produce for each selection. The control UI input and visualization display can be described as 1D, 2D, and 3D. For example, a 3D-2D (3D input-2D display output) primary mechanic might use a 3D control point to produce three points that define a circle in a 2D plane on a visual display.

A very general 2D-2D mechanic uses a 2D input like a finger on a touchscreen to derive a control point for mapping input circles inside the unit disc to output circles inside the unit disc which can then be mapped to a 2D display. The output circles can be found by choosing three input points on each input circle, applying the mapping PHI φ to the input points, resulting in three output points, and using the resulting three output points to solve for the unique output circle that contains the new points. While this is a general solution for zooming and shrinking circles according to PHI φ, it can be subject to floating point error, which may be mitigated in some circumstances by choosing input points intelligently or by using more than three input points.

Many 2D-2D mechanics can be implemented in terms of disc sectors. One way to define a sector is to choose an ordered pair of points on the boundary of the unit disc. The disc sector is bounded by the line segments connecting the points with the center of the unit disc and the boundary arc between the two points. Thus, by applying a disc automorphism to pairs of boundary points, a mapping on disc sectors can be achieved. Note that while disc automorphisms are particularly convenient for this purpose, other boundary point mappings can yield similar results.

2D-1D mechanics take 2D control points and use them to manipulate points in a 1D interval. For simplicity, this interval can be chosen as [0, 1], but any other interval could easily be used. Any function mapping the interval to itself that can be parameterized using a 2D control point can be used to drive a visualization. One application is for zooming parts of the interval whilst simultaneously shrinking others.

Note that although the output of a 2D-1D mechanic is one-dimensional, such a mechanic can be used to display data in more dimensions. For example, a vertical menu as inFIG. 15may be specified based on one-dimensional information describing the boundaries of the menu entries.

2D-0D mechanics use 2D control points to assign a value in the interval [0, 1] to each selection. This can be implemented by using a 2D-1D mechanic and assigning boundary points to each selection. The final value is based on the distance between the transformed boundary points. One embodiment uses this value to determine the opacity of a highlight color for a selection.

3D-3D mechanics use 3D control points in the 3D displayed unit ball which can be a unit sphere along with its interior to manipulate radial conical sections of the 3D unit ball emanating from the origin. The radial conical sections of the 3D unit ball can be analogous to sectors for 2D-2D mechanics. These cones can be given an orientation to distinguish their interior from exterior. Each of these cones can be associated with a sphere tangent to both the boundary of the cone and the boundary of the 3D unit disc. Each cone is also equivalent to a sphere that is tangent to the cone at the same circle at which the cone intersects the 3D unit ball. Planes through the origin represent degenerate radial cones, and they can be associated with a sphere tangent to the 3D unit ball and the plane.

In an embodiment, 3D-2D mechanics can be created by operating on the circles at which radial cones intersect the 3D unit ball.

In an embodiment, 3D-0D mechanics can be created by computing the area/volume of the circles/spheres manipulated by the previous mechanics.

Boundary Pair 2D-1D Mechanic Example

With reference toFIGS. 14A and 14B, the points3001on the unit interval3005are mapped angularly onto the points3002on the boundary of the unit disc3006. The listings3007are associated with pairs of boundary points3002. The disc automorphism PHI φ is used to map boundary points3002to boundary points3003, which can then be transformed angularly back onto the unit interval3005to points3004on the unit interval3005. In some embodiments, the distance between pairs of adjacent points3004can be used to alter the visualization of listings3007. For example, the points3004can be used to determine the opacity of a highlighting layer so that the listings3007associated with points3004that are further apart appear with a greater degree of highlighting. In some embodiments, the points3004can be used to determine the upper and lower edges of menu listings as inFIGS. 52-53.

Satellite Circle 2D-2D Mechanic Example

An embodiment of a “satellite circle” is illustrated inFIG. 6. The “satellite circle”122can be defined as a circle tangent to both the radial sides of a sector of the unit disc123and the boundary of the unit disc123. Because a satellite circle122can be defined in terms of a disc sector, a mapping of disc sectors translates to a mapping of satellite circles122.

FIGS. 6-8illustrate a visualization based on a satellite circle primary mechanic. With reference toFIG. 6, the circular transformations of the satellite circles122can have a specific geometric relationship to the transformation of the central circle121.FIG. 6illustrates a group of twelve satellite circles122that are equal in size and placed between an outer unit disc123and a central circle121that can be graphically transformed through the UI. In particular, because the disc automorphism PHI φ is a continuous mapping of circles to circles, the result of the transformation is guaranteed to have a configuration similar to that depicted inFIGS. 7A, 7B,8A, and8B. As the control point153is moved outward, the satellite circles122closer to the direction of the control point153become larger, and the circles122furthest from the direction of the control point153become smaller.

This transformation of circles can be achieved by taking three points on any circle inside the unit disc123and then transforming those points using the disc automorphism PHI φ. The resulting points lie on the transformed circle, which can be solved for by finding the unique circle containing those three points. This is a brute-force solution that can be used for any of the circles inFIG. 7Aand performed completely in parallel, but it can have issues with floating point error. More accurate results can be achieved for this specific configuration by using two separate parallelizable techniques: one that solves for central circles121, and one that solves for satellite circles122.

FIGS. 7A and 7Billustrate a process for solving for the image of the central circle121under the transformation PHI φ. The inventive system applies the transformation PHI φ to the points151and155that are collinear with both the control point153and the center111of the central circle121. Due to symmetry of the transformation, the images φc(p) and φc(q) of these two points under the mapping PHI φ are still collinear with the center of the transformed central circle121, so they can be used to solve for the center of the circle by averaging their locations.

The circular transformations of the satellite circles122can have a specific geometric relationship to the central circle121. The circle transformation can be achieved by taking three points on any circle inside the unit disc123and then transforming the satellite circles122using the Möbius disc automorphism, and solving for the resulting circle. This solution can be performed completely in parallel, but in general it can have issues with floating point error. The inventive transformation process can use two separate (parallelizable) techniques: one that solves for central circles121, and one that solves for satellite circles122.

In an embodiment, a processor of the computing device running UI software can solve for the central circle as shown inFIGS. 7A and 7B. With reference toFIGS. 7A-7B, the center111is associated with the beginning of a user gesture, and the control point c153is associated with the current position of the gesture. A processor can apply the transformation PHI φ to the points151and155that are collinear with both the control point153and the center111of the circle121. With reference toFIG. 7B, due to symmetry of the transformation PHI φ, the images of the two points151,155under the mapping PHI φ still lie on the same line150as the transformed central circle121, so a processor can use them to solve for the center of the transformed circle131by averaging their locations. A zero-length control point where the location of the control point153is the same as the center point111is a special case where the central circle doesn't move.

FIGS. 8A and 8Billustrate a process for solving for the image of a satellite circle122under the transformation PHI φ. The inventive system can illustrate the movement of the central circle121by solving for a satellite circle122. The inventive system can solve for the locations of the satellite circles122based upon a geometric relationship between the satellite circles122and the central circle121.FIG. 8Aillustrates the satellite circles122displayed touching the unit circle123and central circle121. The transformed satellite circles122remain tangent at the same points. Radial lines extending from the center of the unit circle123through the centers each of the satellite circles122also contacts the unit circle123at the touch points with the satellite circles122. More specifically, the unit circle center111, the centers of the satellite circles122, and the contact points of the satellite circles122with the unit circle123are all collinear. With reference toFIG. 8B, the user can swipe up and right and the central circle121moves to the lower left, the centers of each of the satellite circles122and the contact points of the satellite circles122with the central circle121are also each collinear.

To solve for a satellite circle S124, the unique properties of the transformation φccan be exploited. For example, point p126is a point where the unit circle D123and satellite circle S124touch and point q120is a point where the central circle C121and satellite circle S124touch. The unit circle center111is aligned with point p126and point q120as well as the center of the satellite circle124. As the central circle121moves to the lower left, the center point mc151of the central circle121also moves to the lower left as shown inFIG. 8B. The point φc(p) can still be at the intersection of φc(D)=D and φc(S), and φc(q) can be at the intersection of φc(C) and φc(S). In other words, the transformed circles should still be tangent at the same points. RDcan be a line from the center of the unit disc123to the point φc(p) and RCcan be a line from the center151of the central circle121to the point φc(q). The center of satellite circle S124can lie at the intersection of the lines RD132and RC.134.

With reference toFIG. 9, in general, it may be problematic to solve for the center ms of the satellite circle φc(S)124by intersecting lines RD132and RC134because these lines may either be the same line where the starting point is control point 0 or these lines can intersect at a very shallow angle. Instead, RS132can be a line originating at (φc(p)+φc(q))/2 in a direction that can be perpendicular to φc(p)−φc(q). The center ms of the circle φc(S)124can be found by intersecting the lines RD132and RS134. Given ms, the radius of the circle φc(S)124can be found by computing the distance to one of the points φc(p) or φc(q).

Radial Sensitivity Adjustment

When a larger number of satellite circles are simultaneously shown on the computing device display, it becomes more difficult to precisely zoom in on a specific satellite circle without additional control point preprocessing. The inventive system can have a “Radial Sensitivity Adjustment” which eliminates this zoom resolution issue by ensuring a consistent radial zoom that is independent of the number of satellite circles simultaneously being displayed. The Radial Sensitivity Adjustment is achieved by creating a band along the outer control circle for which zoom is defined in terms of satellite radius. In practice, when dealing with satellite circles, the maximum control point length should be less than one. In cases where the satellite circles are tangent, the distance from the center to the tangent point is a possible maximum control point length.

FIG. 10is a graph illustrating an embodiment of a relationship for adjusting the length of a control point based on a normalized physical input radius. The 45 degree line142represents a linear relationship between the user UI input radius X and the control length Y. The horizontal X movement can represent the physical radial distance from the finger on the touchscreen (or other UI input) to the initial touch point. The vertical Y movement can represent the new length assigned to the control point. In the graph the lines142and144pass through the horizontal line whose distance from the max control length can be defined in terms of satellite circle radius. The line142terminates at the horizontal line that represents the unit disc outer diameter tangent to the outer portions of the satellite circles.

When there are a large number of satellite circles, it can be difficult to control the satellite circle visualization because the unzoomed satellite circles are very small, and the amount of zoom on a single satellite circle for a given input radius is proportional to the number of satellite circles. A radial sensitivity adjustment can ensure that a consistent zoom is achieved independent of the number of satellite circles. In an embodiment, a processor can be configured to provide a non-linear relationship between the user UI input radius X and the control length Y such as the curved line144. In an embodiment, the user can configure the UI to alter the relationship between the user UI input radius X and the control length Y so that as the UI input radius X is greater than 50% of the max input radius, the relationship between the user UI input radius X and the control length Y can result in more UI control sensitivity which can be configured by the user to match the user's sensitivity preference. Once the user has configured the sensitivity, the system can use the input radius interpolate between the line y=x and a line with a more gradual slope depicted inFIG. 10merging with the line144, yielding a higher sensitivity in the radial area of the input space where more UI controls are being performed.

Sometimes a UI display may only require a few satellite circles, so for visual consistency or aesthetic symmetry, it can be necessary to cap the maximum satellite circle radius. With reference toFIG. 11, it can be useful to not have enough satellite circles to completely fill the annular space between the unit disc and the central circle. In the illustrated example there are only four satellite circles122which must each fit between the central circle121and the unit circle123. The processor can display the satellite circles122in an evenly distributed manner in the annular space between the central circle121and the unit circle123. This results in satellite circles122being displayed in a manner where the satellite circles122do not touch one another. The reduced number of satellite circles122can be manipulated using the same disc automorphism described in embodiments having more satellite circles122.

Boundary Pairs Examples

In some embodiments, the UI system can perform boundary pair mechanics. With reference toFIGS. 12 and 13, a boundary pair mechanic can be thought of as operating on disc sectors2001which can be pie slice shaped pieces arranged in the unit disc2003. InFIG. 12the unit disc2003has eight disc sectors2001which are uniform in size and shape and each of the sectors can have spaced boundary points2002on the boundary of the unit disc2003. In this example, the user has initiated a gesture resulting in a control point at the center of the unit disc2003and gestured outward resulting in control point2004between boundary pair points X and Y. With reference toFIG. 13, this radial gesture can result in the movement of boundary points2005on the circumference of the unit disc2003. The UI system interprets this gesture as expressing interest in the sector determined by the boundary pair points X and Y which results in this disc sector2001expanding while the other disc sectors2001are compressed. Boundary pair mechanics apply a disc automorphism to pairs of points2002on the boundary of the unit disc2003, resulting in a new pair of points2005on the unit disc boundary for each pair2002to which the transformation is applied.

With reference toFIGS. 14A and 14Ban embodiment of a transformation PHI φ of a data listing is illustrated. In the illustrated embodiment the data listing shown on the visual display can be uniform. The selections3007on the vertical listing are A, B, C, D, E, and F which can each be associated with pairs of points3001inside the unit interval3005between 0 and 1. The unit interval3005can be mapped angularly onto the boundary of the unit disc3006, and the same mapping can be used to associate the selections3007A, B, C, D, E, and F with arcs bounded by the points3002on the unit disc3006. The user can manipulate the control point3003via a gesture whose beginning can be associated with the center of the unit disc3006.

The UI can apply the disc automorphism PHI φ to the unit disc3003, which yields the transformed points3003. The portion of the boundary containing the transformed points3003can be mapped back to the unit interval3005such that the angular distances between each relevant pair of points3003is proportional to the distance between the associated pair of points3004. The final result points3004can be used to drive a visualization such as a menu zoom. In some embodiments, the distance between result points3004can be used to drive a visualization such as the intensity of the highlight color for menu selections.

Angular Selection Mapping

SeeFIGS. 15-17for an example of an “angular selection mapping.” In some embodiments, any visualization that operates with a scroll mechanic such as the scroll disc123shown inFIG. 16can create an angular selection mapping control disc125shown inFIG. 17. This mapping associates bounding angles with each selection. Adjacent selections could share bounding angles. These angles are not required to lie between 0 and 2π. Only selections with at least one bounding angle in the current selection window are intended to be visible. In this example, only the areas associated with h to m are displayed.

As in other embodiments, the control point is used to manipulate a visual display output. An example of a visual display is illustrated inFIG. 15. In this example, there is a vertical arranged data list162with each row of the data list being labeled “a” through “n”. A visible window164identifies the listings that are displayed on the visual display. The other listings are not displayed which can be virtually above or below the visible window. In this example, the listing data h through m are in the visible window164and displayed on the display of the computing device.

Rather than displaying each row of data in the same text size, the rows of data that are in the visible window164can be displayed in different text sizes on the display so that the listing of interest which can be the center displayed listing has the largest display area or fully enlarged display status. The adjacent listings can also be displayed in an enlarged format that is smaller than the fully enlarged size. The peripheral listings at the upper and lower portions of the display can have decreasingly smaller text size formats based upon the distances from the center fully enlarged listing.

InFIG. 15, the center listing “k” is fully enlarged text “do” at the center listing of the visible window. The first adjacent listings are “j” above and “l” below which correspond to the partially enlarged text “sed” and “eiusmod” that can be displayed in an enlarged text but a text size that is smaller than the center listing. The next outer listings that are still within the visible window are “i” above and “m” below which correspond to the text “elit” and “consectetur” that are in a smaller text size. The final row in the visible window164is “h” at the top of the displayed visible window which can be in the smallest text size.

FIG. 16illustrates a scroll disc128that corresponds to the visualization inFIG. 15. The corresponding scroll disc128that is illustrated has a scroll selector which is divided into six areas that correspond to the listings h, i, j, k, l, and m. In the illustrated example, the control point213is in the k area of the scroll selector which corresponds with “do” being fully enlarged in the visible window shown inFIG. 15. The user can swipe in a clockwise or counterclockwise direction and the control point213can move through different scroll zones. With reference toFIG. 17a control disc130is illustrated, which can be controlled to alter the listing shown in the visible window (shown inFIG. 15). The user can manipulate the control disc130by swiping in a counterclockwise rotational movement to move the data list162up or in a clockwise rotational movement to move the data list162down. As the data list moves, the six rows of text data in the visible window will scroll and the fully enlarged text can shift up or down changing the text which is fully enlarged and partially enlarged in the visible window. Thus, as the user swipes, the listings will also change the data listings in the visible window.

The distribution of the data listing can correspond to the circumference of the control disc130. In the illustrated example, the listing can include 14 rows of text data and the control disc130can be divided into 12 equal angular segments. If the entire control disc130is 360 degrees or 2π then each of the 12 angular segments is n/6. Table 1 below lists the marked letters, angular position, and visible status inFIGS. 15-17.

The scroll point can be mapped to the control disc130by finding the ψ-rotated scroll point's location in the φ-rotated control disc. The rotation φ corresponds to a Rotational Sensitivity Adjustment which is described later. The visible window can be found by matching the angles subtended by the scroll selector with the angles corresponding with selections on the φ-rotated control disc. In order to manipulate the visible window, a boundary pair mechanic can be applied to the pairs of boundary points bounding the selections in the scroll disc123. The visible window can be frozen when a circular scrolling gesture continues such that the visible window would otherwise move beyond the beginning or the end of the list of selections.

Scroll Mechanic

The inventive system and method can provide additional different UI mechanics for making user selections. In an embodiment, the present invention can be applied to a more familiar rectangular data layout displayed on the computing device display. The data can be arranged in a table with vertical or horizontal data cells on the visual display. The UI system can also provide a means for scrolling through content that can be temporarily visible on the display where the user can display some data and then scroll to display data that is concealed and not displayed. A scroll mechanic is a technique for determining a contiguous range of active selections from a larger group of selections. This allows for placing an upper bound on the number of elements that the primary mechanic is applied to, and in some embodiments inactive selections will not be displayed at all.

FIG. 18illustrates an embodiment of a “scroll disc”128used as the input controls for the UI system for scrolling control. The scroll disc128can have different designated scroll control regions. Alternative embodiments might use some combination of rotations or reflections of the same configuration. In this example, the circular region123can be associated with a touch circle on a touchpad, but any other 2D input device could be similarly associated. The center of the unit disc123can be a “neutral zone”201. A half annular space on the right side of the unit disc123to the right of the neutral zone201can be a scroll selector area203. An area to the left side of the unit disc123to the left of the neutral zone201can be a “back selector” area205. The user can touch and swipe to manipulate a control point on the circular unit disc123to control the UI.

With reference toFIG. 19, a user has dragged the touchpad resulting in the control point213. Since the control point213is in the scroll zone223, the control point213is also a scroll point215. The back point211is a user input that causes the processor to cause the display to reverse prior visualization in the displayed information visualizations, and the scroll point223is a user touch point used as the control input to scroll visualizations. When the control point213is in the scroll zone223, the back point211is at the origin center of the scroll disc123. An upper back zone225and a lower back zone227are to the left of the back point211and the vertical center line of the unit circle123.

From the control point213, the user can perform scrolling through a listing of data where some of the listing data is clearly displayed and at least some of the listing of data is not displayed. When the control point213is in the scroll zone223, the user can swipe upward in a counterclockwise direction to cause the display to scroll the displayed data in one direction or swipe downward in a clockwise direction to cause the display to scroll the displayed data in an opposite second direction. For example, swiping upward in a counterclockwise direction causes the listed data to scroll upward and swiping downward in a clockwise direction can cause the listed data to scroll downward.

With reference toFIG. 20, the user has dragged to the lower left of the position at which they touched the touchpad, and a processor has assigned a control point213in the lower left back zone227. The “back point”211is the projection of the control point213onto the horizontal axis, and the scroll point is the projection of the control point onto the vertical axis. The back point211is used to control the back visualization. The back point211can be a value between zero and one, which can be thought of as a point along the negative x axis to the left of the origin211.

Both the control disc and the scroll disc can have an associated disc rotation. The control rotation can be represented by ϕ and the scroll rotation can be represented by ψ. When the control point is inside the scroll selector, the control disc's rotation can be controlled according to rotational sensitivity adjustment the rotational sensitivity adjustment (RSA) which will be described with reference toFIGS. 36-38below.

FIGS. 21-23illustrate an example of a user interaction with an embodiment of a scroll disc128. InFIG. 21the user has touched the touchpad and dragged outward, resulting in the control point213in the scroll selector area123. The user then drags the control point left toward the upper scroll margin207. If the control point213moves from the scroll selector area123onto the boundary of the scroll margin207, the interior regions of the scroll disc128including back selector205, scroll margin207, neutral zone201, and scroll selector203can rotate so that the control point213stays on the boundary. In some embodiments, rotation of the scroll disc can inactivate the back selector for the duration of the gesture. With reference toFIG. 22, the user's finger has swiped left and the control point213is moved onto the vertical line dividing the scroll margin207from the scroll selector123. With reference toFIG. 23, as the user continues to swipe the control point213to the left, the interior zone areas have rotated counterclockwise within the scroll disc with the control point213on the center dividing line between the scroll margin207and scroll selector123.

FIGS. 24-26illustrate another example of a user interaction with an embodiment of a scroll disc128. With reference toFIG. 24, the user has dragged the control point213into the scroll margin207from below, bypassing the boundary between the scroll margin207and the scroll selector123. The user drags the control point counterclockwise away from the scroll selector area123. InFIG. 25, the control point213remains in the scroll margin207while the finger and control point213continue to rotate counterclockwise. All of the interior regions of the scroll disc128including back selector205, scroll margin207, neutral zone201, and scroll selector203can also rotate counterclockwise, but at a faster rate than the control point, allowing the boundary between the scroll margin207and the scroll selector123to partially catch up to the control point. The back selector205may become deactivated due to the rotation. InFIG. 26, the control point213has continued on a counterclockwise path, and the line between the scroll margin207and the scroll selector123has caught up to the control point213.

With reference toFIGS. 27-29, the user drags the control point213clockwise while the scroll disc128and the interior regions including the back selector205, scroll margin207, neutral zone201, and scroll selector203remain stationary. With reference toFIG. 27, the scroll disc128has been rotated counterclockwise counterclockwise and the user is in the middle of a clockwise drag, with the control point213in the scroll margin207. With reference toFIG. 28, the user continues to drag the control point213clockwise from the scroll margin207into the scroll selector123. With reference toFIG. 29, the user has continued to drag the control point213clockwise into the scroll selector123.

Jumping Scroll Mechanic Example

In this embodiment, the active window for the scroll mechanic jumps as the control point crosses a scroll boundary. This is useful in embodiments such as color zoom discussed below. This example uses a touchscreen, but any two-dimensional input device could function similarly. With reference toFIG. 30A, the user starts by touching a touchscreen at an initial point associated with the control point5001and the UI computes a touch circle around the center control point5001, with the selections A, B, C, and D5004being active meaning the control point drives their visualizations on the display of the computing device. The user can swipe on the UI touchscreen downward into the selection “D” and then swipes clockwise to control point5003. When the gesture crosses the scroll boundary5002, the active window jumps, and the selections E, F, and G5005become active as shown inFIG. 30B. In this example, the user has stopped and released on selection F and the UI can interpret this gesture as the user selecting the selection F.

Continuous Scroll Mechanic Example

The inventive UI system can be used for scrolling through a list where only a portion of the list can be displayed on the display of the computing device. InFIG. 31A, a listing of data includes an active portion7010that is displayed and an inactive portion7012that is not displayed. The user has touched the touchpad, resulting in an initial control point7000and touch circle7005has formed around the initial control point7000. The user then swipes in a downward counterclockwise path to control point7001. In this example, the listing has nineteen lines of data. The top seven lines of data are the active portion7010and the bottom twelve lines of data are the inactive portion7012.

InFIG. 31Bthe user has continued to swipe in a clockwise arc from control point7001to control point7002. And the UI can respond to the swipe from7001to7002by scrolling the active portion7010of the listing of data down.

Centered Scroll Mechanic Example

With reference toFIGS. 32A-32C, an embodiment keeps the visible active window7110centered angularly around a control point. The UI detects the beginning of a user gesture and computes a touch circle. The detected gesture continues to the point7101, which is used by a processor to compute the two points that lie on the boundary of the touch circle and form a right angle with the point7101at the center of the touch circle. The angles of the two computed points further determine an active window7110in the list of selections7112. The active window can be displayed using a visualization driven by the point7101. InFIG. 32B, the detected gesture continues to the point7102, allowing a processor to detect a new pair of points on the boundary of the touch circle, and thus a new active window7110in the list of selections7112. InFIG. 32C, the detected gesture continues to the point7103, allowing a processor to detect a new pair of points on the boundary of the touch circle, and thus a new active window7110in the list of selections7112.

Grouping Mechanics

In an embodiment, the UI system can perform grouping mechanics as a way of subdividing a larger list of concrete selections so as to reduce the number of selections from which the user must choose at any one time. A predefined “chunk size” is used as a guide for the maximum number of selections that can be chosen from at one time. Based on this chunk size, a grouping mechanic introduces a tree of intermediate subgroups, with the concrete selections as leaves of the tree. Grouping mechanics can be combined with scroll mechanics to allow users to traverse between sibling subgroups without going through the parent subgroup. See the color zoom section below for an example.

In some embodiments, the following rules can be used to partition a given list of selections of length L, into a tree of nested sublists using a given chunk size c. A target height h, for the tree can be found by taking the ceiling of the logarithm of L base c. Compute a target child subgroup size, x=L/round (L{circumflex over ( )}(1/h)). The current subgroup should contain L/x children (either subgroups or leaves). Define the boundary selection index, p, as n−L mod x. If a child has a selection index less than p, it is a subgroup containing x elements (or it is a leaf if x is 1). Otherwise, the child has a selection index greater than or equal to p, and it is a subgroup containing x+1 elements. This same operation can be applied recursively to each subgroup to form the tree of sublists. The sublists can be UI selections as described in other embodiments, so that a user can manipulate visualizations of the sublists on a display using gestures with radial and/or angular components. By utilizing a selection mechanic the user can move deeper into the tree away from the tree's root; by utilizing a scroll mechanic, the user can move laterally within the tree; and by utilizing a back mechanic the user can move upward in the tree toward its root.

A transition mechanic is a way of using a control point to transition from using a static layout when the user is not interacting to using a dynamic one as the user interacts. This can be achieved by interpolating based on the length of the control point. Given a maximum length m (possibly an application setting), the length of the line segment joining the origin and the control point divided by m is used to compute an interpolation factor k. The interpolation factor k is used to interpolate between a static location and the corresponding dynamic location computed by a primary mechanic. Thus, when the length of the control point is zero, the layout remains static, and as the control point moves away from the origin, the effects of the dynamic transformation become dominant. Other embodiments can trigger an animation from static to dynamic when the beginning of a user gesture is detected by the processor, and trigger the reverse animation when the end of a gesture is detected. A more complicated embodiment uses animations that can be accelerated based on the length of the control point. When the UI detects the beginning of a user gesture it can start an animation from a static visualization to a dynamic visualization, and the previous interpolation technique can also be used to compute an interpolation. Whichever results in the dynamic location being more dominant wins, so a user moving quickly does not need to wait for animations to complete.

Transition Mechanic Example

FIGS. 33-35illustrate an example of a transition mechanic. This example embodies a UI touchpad controller that adds a transition mechanic to a basic satellite circle primary mechanic. InFIG. 33the letters105are positioned in a static layout that is visible even when no touch input is present. A finger101has made an initial touch, which the processor interprets as control point111with no visible effect on the displayed letters105. With reference toFIG. 34, as the finger101touch moves away from the initial control point111, the satellite circle primary mechanic becomes more dominant, until inFIG. 35the satellite circle primary mechanic is completely dominant. InFIG. 35the letters107are shown in satellite circles122between the central circle121and the disc123.

While the letters105inFIG. 33are illustrated in a table with rows and columns, in other embodiments, the two-dimensional array can be any other organization of content in a two-dimensional configuration such as arrangements of information in any geometric shape or a custom layout. In the illustrated embodiment, the UI uses a “satellite circle primary mechanic” which allows combining radial and circular swipe movements to help a user select a specific letter which speeds up and/or improves the user's data selection process by interactively zooming over the letter options in the satellite circles122. In this example, there is one central circle121that represents a context or label, and there are multiple satellite circles122representing individual and distinct user choices or selections which surround the central circle121. The visualization can place any sort of content data inside these satellite circles. It is also possible to nest this visualization inside satellite circles122to represent multiple levels of data selections.

With reference toFIG. 33, when the user isn't interacting with or touching the touchpad, the letters “a” through “l” can be displayed in a more traditional non-circular two-dimensional array display layout105on the visual display. The user can touch the touchpad portion of the UI and the initial touch point111can define a center of a circle115. When there is the initial contact of the finger101with the touchpad UI, the letters are displayed in a non-circular two-dimensional array display layout105.

With reference toFIG. 34, the finger101moves from the initial touch point111in an outward direction towards an upper right portion of the circle115on the touchpad. The non-circular layout of the letters shown inFIG. 33transforms into the rotary circular layout as a result of user interaction. In this example, the finger101swipes outward and the displayed letters “a” through “l” are shown in a transitional configuration106as the display transforms from a two-dimensional array into a circular arrangement of letters. InFIG. 34, the letters106are in a partially transformed state with the adjacent letters not in alphabetical order. If the user's finger101is removed from the touchpad, the displayed letters “a” through “l” will return to the non-circular two-dimensional array display shown inFIG. 33.

With reference toFIG. 35, the user's finger101has moved in a straight line to a position close to the circle115. In response to the user's finger101movement on the touchscreen UI, the letters “a” through “l” are each placed in an individual satellite circle122and the satellite circles122are adjacently arranged in a circular pattern between an outer unit disc123and a central circle121. The position of the central circle121within the unit disc123can depend upon the position of the swipe movement. In this example, since the finger has moved outward from the initial touch point111on the touchscreen UI and the system responds to this finger position by moving the central circle121towards the lower left within the unit disc123in the opposite direction of the upper right swipe movement. The central circle121is not concentric with the unit disc123and the circles122have a diameter based upon the position within the unit disc123. InFIG. 35, the finger101moved towards the letter “c” and the letter “c” is the largest displayed letter. The displayed letters “a” through “l” have different smaller sizes which are smaller in size than the letter c and are located adjacent to each other between the central circle121and the unit disc123. The letter “j” is the smallest letter because it is opposite the letter c in the circular letter configuration.

The described UI can be used to enlarge and select a specific satellite circle. InFIG. 35, the user has used the UI to manipulate the displayed information and the satellite circle122containing the letter c is enlarged. The user can then use the UI to select the letter c. The UI can respond by inputting the letter c or if a listing of options which can be labeled as a through l. If the user selects c, the system can select the listing associated with the letter c. The UI input system can then reset and the user can use the UI to enlarge and select another satellite circle which can be a letter. In this manner, the UI can be used to continue to input text, numbers, or any other data that can be associated with the satellite circles.

From the finger position shown inFIG. 35, the user can then swipe in a circular direction in clockwise or counterclockwise directions around the circle115. If the user swipes in a circular motion around the unit disc123, the largest satellite circles122and corresponding largest letter will shift sequentially with the location of the touch point. For example, if the touch point moves in a clockwise swipe, the largest letter will switch sequentially in the sequence: c, d, g, h, l, k, j, i, e, f, a, b. Conversely, if the user touch point swipes counterclockwise, the largest satellite circles122and corresponding largest letter will switch sequentially from c, b, a, f, e, i, j, k, l, h, g, d. The size variations of the satellite circles122and letters can be based upon the relative positions of the central circle121and the unit disc123. As discussed, if the user moves the finger101to the initial touch point111, the central circle121can be concentric to the unit disc123which results in each of the satellite circles122and corresponding letters being equal in size. One embodiment attempts to minimize the total angular distance between the locations of selections in the non-circular layout shown inFIG. 33and their locations in the rotary layoutFIG. 35.

Back Mechanic

A back mechanic is a way of transforming a control point in the unit disc or ball into a 1D “back point” lying in the interval [0, 1]. A back mechanic requires storing a history of prior control points representing successful selections. Although these control points may be the same as those present at the time of selection, a more canonical representation may be chosen at selection time (and the associated visualization updated accordingly). The back point can be detected as described inFIG. 19andFIG. 20and then applied to decrease the length of a most recently stored control point, affecting its visualization assuming one is visible. As discussed above with reference toFIGS. 18-20, the back mechanic can be implemented by touching to set a control point in the neutral zone and then swiping to the left across the scroll zone/back zone boundary. Other embodiments might detect back gestures as moving generally to the right or up or down. The UI can respond to a detected back selection reconfiguring for further display and UI input based on stored historical data.

Rotational Sensitivity Adjustment

Some primary mechanics that depend on a 2D control point may give users extra angular resolution for making selections by making an angular adjustment. Rotational Sensitivity Adjustment (RSA) ensures a minimum level of angular accuracy UI control. This means that as a user manipulates a control point angularly, selections move into and out of focus at a manageable rate, even when there are a large number of them.

The RSA can operate on the control disc mechanic so that no angular drag should be applied in the neutral zone of the unit circle defined by the outer boundary245circle as shown inFIG. 36. With reference toFIGS. 36-38, at the inner boundary, an Angular Drag gesture can be applied continuously starting at origin control point240in the neutral zone241and swiping the control point to a point within the inner boundary243which can be the inner diameter of the scroll selector250as shown inFIG. 36. InFIG. 37the user swipes the control point clockwise along the inner diameter243and the scroll selector250does not rotate. As shown inFIG. 38, the clockwise swipe can continue in rotation progressing to 180 degrees from the start at the top of the inner boundary243to the bottom of the inner boundary243of the scroll selector. Again, because the control point swipe is within the inner boundary243the scroll selector250does not rotate.

In contrast, with reference toFIGS. 39-41, the user begins a gesture with a control point at the origin control point240and swipes outward to the outer boundary245of the scroll selector as shown inFIG. 39. The user can then swipe clockwise as shown inFIG. 40and the scroll selector250can also rotate clockwise with the control point so that the control point moves through exactly n regions in a revolution, with n being a sensitivity setting. In this example, the control point251has moved through π/2 radians and the scroll selector250has rotated clockwise 3 selections which can be π/4 radians. The user can continue to swipe the control point251180 degrees clockwise as shown inFIG. 41. The maximum RSA adjustment can occur along the outer boundary245. In the illustrated example, a gesture subtending7C radians moves through six selections.

Combining Mechanics

The various mechanics and visualizations described herein can be combined to create a concrete input mechanic. This typically manifests as rules for preprocessing control points before they are given to the primary mechanic, rules describing how to obtain a back point from a control point if a back mechanic is present, and rules for determining the visible selections and the data associated with them (e.g., circle coordinates or sector boundary pairs).

Combined Mechanic Example

One embodiment combines the satellite circle primary mechanic with a back mechanic. With reference toFIG. 72, an array of satellite circles811is arranged inside a unit disc810surrounding a central circle812. No satellite circles are arranged on the left side of the unit disc in order to make room for a back mechanic. The scroll disc inFIGS. 18-20can be used to drive a satellite circle mechanic using the scroll point and a back mechanic using the back point. Releasing in the scroll selector can perform a selection, and releasing in the back selector can perform a back action.

Context Menus

An input mechanic can use a special gesture input to trigger a contextual visualization, which can be a menu. This gesture resembles the gestures used for other input mechanic embodiments, except for the way it starts, which may use a “gesture prefix” which can be a predefined UI input. For example, a gesture prefix could be defined as a double tap on the touchpad sensor. Once the gesture prefix is detected, the UI can recognize this input and initiate context menu controls and the UI control point can drive a visualization associated with the context menu. If nothing is selected during a context menu interaction, the control point animates back to zero which makes the context menu of the UI go back to a default rest state, and the context menu can be deactivated. Future gestures without the requisite prefix will associate their control points as they did before the context menu was invoked.

Page Zoom Visualizations

With reference toFIGS. 42A and 42B, the inventive UI system can be used as a page selector that allows the user to efficiently select where in the document they want to start, no matter how long the document is using the described scrolling methods. The further the user zooms out, the smaller the page will appear on the visual display and conversely, the further the user zooms in, the larger the page will appear on the visual display.

With reference toFIGS. 42A and 42B, page zoom allows for zooming in on a large amount of information such as a large document to select a region of the large document in which the user wants to work. The UI can be configured through a user setting with a predefined “page size” and the displayed document can be divided up into a plurality of pages6001of overlapping content based upon the page size. In an embodiment, the first half of a second page6004might contain the same content as the second half of the first page. The half page overlap is arbitrary and could be any interval. This overlap of pages allows users to choose a page that does not end in the middle of the content of interest and two different pages will generally contain the same content of interest. Thus, if there were no overlap, a user might need to focus in an area that is located at a split between adjacent two pages, which is not ideal.

InFIG. 42A, points6002representing heights on the display are mapped to points6003on the boundary of the unit disc, and each of the selections6001are associated with interleaved pairs of points6003. InFIG. 42B, a transformation can be applied on the boundary of the unit disc based on the movement of the swiping of the control point6005to control point6009, resulting in the redistribution of points6008on the boundary of the unit disc. These points6008can then be transformed back to the points6007based on their angular distribution. Pages are sorted such that6006appears on top, its neighbors are underneath it, their neighbors are underneath them, etc. For example, the pages can be sorted from top to bottom, E6006, F6017, D6015, G, D, C, B, A . . . . This UI configuration ensures that there will be no discontinuity when a new page gains focus because the overlapping content that is shared between the new page and the page that is losing focus as the UI transitions to the new page is identical. One drawback to this configuration is that some content can be obscured behind other content when it is not in focus.

With reference toFIGS. 44A and 44B, an embodiment of a page zoom UI system is illustrated. The UI system can display data in a plurality of individual pages which make up a document that is being displayed. The pages can be displayed at a “page size” which can be a predefined size and/or can be configurable through a user setting and can be based upon the screen size of the computing device. The document can be displayed in a configuration divided up into pages of overlapping content. In the illustrated example, the first half of a second page6204might contain the same content as the second half of the first page. The page overlap can be arbitrary and/or set by a user through UI preference controls. This displayed page overlap helps users choose a page displayed that does not end in the middle of the content of interest. In other words, if there were no displayed page overlap, a user might need to focus in an area that is split between two pages, which is not ideal.

The points6209can initially be evenly distributed along the height of the screen and mapped as points6202which can be uniformly distributed on the boundary of the unit disc. Each of the selections6201around the unit disc can be associated with interleaved pairs of points6209at the top and bottom of the left edge of the selections6201. A transformation of the page displays can be applied on the boundary based on the control point6205swiping to6206inFIG. 44Bresulting in the distribution of points6203on the boundary of the unit disc. These points6203can then be transformed back to the points6210based on their angular distribution. The most focused page6207is displayed with the associated points6210bounding it above and below, and the other pages are displayed such that the interior point6210and its closest neighbor6210bound half of the page. Pages are sorted such that page “E”6207appears as though it is on top, its neighbor pages “F”6210and “D”6208are underneath it, their neighbor pages C and G are underneath them, etc. The obscured half pages represented by dashed lines that are underneath the adjacent pages should not be shown. For example, in this illustration, page E6207obscures the lower portion of page D6208and the upper portion of page F6210. Note that this strategy ensures that there will be no discontinuity when a new page gains focus because at that instant the overlapping content that is shared between it and the page that is losing focus is identical. More specifically, the upper half of page E6207can have the same content as the lower half of page D6208and the lower half of page E6207can have the same content as the upper half of page F6210, so the obscured content is actually displayed in page E6207and no information is not visible. This ensures that all of the content is always visible regardless of overlapping pages.

With reference toFIGS. 45A and 45B, points are assigned surrounding each half page vertically and mapped to points6306on the boundary of the unit disc. The swiping from the control point6304inFIG. 45Ato a control point6305shown inFIG. 45Bresults in the UI performing the zooming transformation. In this example, the text data F6303is most enlarged with the neighbor page E above and page G below displayed in a less enlarged format. The text data D is then displayed above page E and text data H is displayed under the text data G in a smaller format. The described zooming based on the swiping of a control point can be used in combination with a highlight effect based on the same mapping.

In some embodiments of the page zoom, the UI can snap the current control point to a new control point in between pages, either immediately or via an animation. This snap feature effectively zooms both half-pages equally resulting in the UI displaying the full page in a uniform manner.

FIGS. 54 and 55illustrate a section zooming function that is similar to the zooming feature described above with reference toFIGS. 44A, 44B, 45A, and 45B.FIG. 54illustrates a listing of text data which has a very small text size. The inventive UI can be used to scroll through the listed data and used to zoom in on the desired text to a text size that can be easily read and possibly edited. The selected text shown inFIG. 54in a fixed text display window that can be traversed by the described scrolling process where the control point is moved angularly. The zooming of the selected text can be achieved by radial outward movement of the control point as shown inFIG. 55. In an embodiment,FIG. 54can be similar to the page data UIs illustrated inFIGS. 42A, 44A, and 45Ain an unzoomed format andFIG. 55can be similar to the UIs illustrated inFIGS. 42B, 44B, and 45Bwhere the page data of interest to the user is selected and zoomed through the inventive UI.

FIGS. 43A and 43Billustrate an embodiment of a listing zoom graphical transformation. When page zoom is applied to text and long lines are present that cannot be wrapped, they can be compressed horizontally as inFIG. 43Bso that the longest line visible on the half-page extends the full width of the page rather than running off the page to the right. When horizontal compression is present, a similar horizontal page zoom similar to vertical page zoom can be applied using horizontal pages to arrive at a focused horizontal page. Variations on this technique exist such as starting with the first horizontal page already zoomed. Horizontal page zoom can also be interleaved with another mechanic. For example, a vertical page zoom mechanic can be used to select a page from a document, with first horizontal page zoomed by default, then a color zoom to select a line, and then, if the line extends beyond the horizontal page, a horizontal page zoom can be used to select a horizontal page.

FIG. 43Ais an unzoomed full document which has four groups of data where each of the groups is four lines of data each between two marker points6101.FIG. 43Bis a zoomed page of a middle portion of the unzoomed document with two groups of four lines of data each between marker points6101. InFIG. 43Athe longest line6105extends almost the full width of the page and inFIG. 43Bthe longest line6105visible on the half-page extends the full width of the zoomed page. Thus, the vertical compression and the horizontal compression can be independent of each other. When horizontal compression is present, a similar horizontal page zoom can be applied using horizontal pages to arrive at a focused horizontal page. In this example, the full page shown inFIG. 43Adisplays sixteen rows of text data with the fifth row6103having four blocks of text data a, b, c, and d. InFIG. 43Bshown row6104is the top row of text data a, b, c, and d. In an embodiment, the UI can be configured to zoom horizontally to a default where the longest row6105is shown extending across the entire width of the display which can therefore limit the horizontal zooming of the UI. In other embodiments, the horizontal zooming may not be limited by the longest row of data. In these embodiments, the longest row6105can be broken into two (or more) lines of text that can be shown in multiple rows if the longest row is substantially longer than the next longest row of data.

InFIG. 43B, the zoomed page has been enlarged to show eight rows of text data in the vertical display in two groups which are each between two marker points6101. Horizontal page zoom can also be interleaved with another mechanic. For example, a vertical page zoom mechanic can be used to select a page from a document, with the first horizontal page zoomed by default, then a color zoom to select a line, and then, if the line extends beyond the horizontal page, a horizontal page zoom can be used to select a horizontal page.

In other embodiments, when long lines of data are displayed that cannot be wrapped, the long lines of data can be compressed horizontally so that the longest line visible on the half-page extends the full width of the page. More complex UI rules are also possible, such as compressing the displayed data according to the longest line in the two most selected half pages. When horizontal compression is present, a similar horizontal page zoom can be applied using horizontal half-pages to arrive at a focused horizontal page. Again, a user data selection UI mechanism can snap the control to a control in-between two half pages. Horizontal page zoom can also be interleaved with another UI mechanic. For example, a vertical page zoom mechanic can be used to select a page from a document, with first horizontal page zoomed by default, then a color zoom to select a line, and then, if the line extends beyond the horizontal page, a horizontal page zoom can be used to select a horizontal page.

With reference toFIGS. 46A-46C, an embodiment of a horizontal page zoom feature is illustrated. InFIG. 46Athe text is obscured because it is displayed with an incorrect aspect ratio where the text appears much narrower than intended by the designer. InFIGS. 46B and 46C, the UI is being used to perform a horizontal zoom so that the text in the grid can be read. A portion of the grid has been expanded467to show a zoomed version of the text a, b, c, and d with each letter in a horizontally adjacent grid box. A processor has detected the beginning of a gesture and computed the corresponding control point461, and the UI has responded by computing a touch circle463. In a method similar to that illustrated inFIG. 44B, the points465are associated with “horizontal pages” of text in an interleaved manner, where at rest, each horizontal page spans three of the points depicted along the top and bottom of the grid. Once zooming is applied, the most focused page is bounded on the left and right by two of the points depicted along the top and bottom of the grid, with another point in between. Similar toFIG. 44B, the other pages are displayed such that the interior point and its closest neighbor bound half of the page horizontally. The right side of the touch circle463has seven markers465which are mapped to groups of vertical lines in the grid467. In the illustrated example, there are four columns between two adjacent markers465. InFIG. 46B, the user has swiped outward between two of the markers465and the UI responds by adjusting the grid467to widen the columns that are mapped between the two markers465. With reference toFIG. 46C, the user can continue the gesture outward until the control point469is closer to the touch circle465and the text in the expanded columns becomes more clearly visible on the display.

With reference toFIGS. 47 and 48, in different embodiments, the inventive UI control system can be used for a plurality of different functions including: file browsing, page selection, cursor selection, local cursor selection, and as a keyboard input for the computing device. FIGS.47and48are flowcharts showing how the different UI functions or different UI modes can be performed sequentially to accomplish the desired task. With reference toFIG. 47, a user may need to edit a document and the UI can first be used as a file browser and the UI can have an input for selecting a desired file. The UI can then be used to select a page from the file for editing and/or review and the UI can have an input for selecting a desired page. The UI can then provide a UI cursor for selecting specific parts of the page for review and/or editing. Once the desired data in the page is selected, the UI can provide a keyboard for performing editing. The user can edit the text and/or data on the page. The UI can provide toggling or switching between a local cursor select for identifying additional local text/data editing and a keyboard for performing the edits. Once the editing is completed, the UI can go back to the file browser functionality and the described process can be repeated. In some embodiments, the files may not have a large number of pages and the UI may not require a page selector function.FIG. 48illustrates another flowchart which is very similar to that ofFIG. 47but does not include the page selector function. The UI's file browser can be used to locate the desired portion of the file and the UI can switch to the cursor select and the selected data can be edited by the local cursor select and the keyboard functions as described. Each of the file browsing, page selection, cursor selection, local cursor selection, and a keyboard input function for the computing device will be described below.

Color Zoom

A 2D-0D primary mechanic can be used to assign intensity to selections. In one embodiment, color zoom can be used to place a cursor on a page of text. At a bare minimum, the most selected selection must be tracked and ideally indicated when focused via some means such as a highlight color change.

FIGS. 49A-49C, 50A-50C, and 51A-51Dillustrate examples of a UI which can focus on areas of interest in organized data. In these embodiments, the UI can apply a zoom mechanic along with a grouping mechanic and a back mechanic. A focus highlight can be used to indicate degree of focus using opacity or text colors or other visual contrasts. With reference toFIG. 49A, a user can use the UI input to set a control point491and the UI can compute a touch circle493around the initial control point491. The touch circle493can have a plurality of segments494which represent each line492of the data on a display and each of these lines can be illustrated in the same uniform manner without highlighting or emphasizing any individual line. With reference toFIG. 49B, the user swipes the control point495outward into one of the segments492. In this example, the second highest line499is highlighted or remains in the original color or shade. The adjacent lines498, which in this example are the highest and third highest lines, can have a lower level of highlighting and the remaining lines492have lower highlighting. The UI can respond by changing the highlighting levels of the data lines that are further from the segment area of the swipe control point495and they can be more out of focus or become less opaque. With reference toFIG. 49C, the swipe control point496has extended to the touch circle493and this can be interpreted as a user selection where only the second line499is highlighted. The highlighting can be done by the UI with an underlay, color, opacity, visual effects such as blinking data, etc.

In some situations, there may not be enough lines for the grouping mechanic to introduce intermediate groups, so the UI can allow the user to select a line499directly. With reference toFIGS. 50A-50C, the line499may have too many characters for a single selection using the UI, so the UI may introduce intermediate groups. an embodiment for using the UI for focusing on a group of characters in line499. InFIG. 50A, the data is arranged in a single horizontal row and the UI is in a no focus mode where all portions of the line499are equally highlighted. The user has begun a gesture using the UI input to create a control point505and a touch circle507has been formed with three segment areas where each segment area corresponds to three different intermediate groups511,512,513of data from line499. The upper right segment of the touch circle505may correspond to the first (left) third of data511, the middle right segment may correspond to the middle third of the data512and the lower right segment can correspond to the last (right) third of the data513. With reference toFIG. 50B, the user has swiped the control point506down and to the right to a lower right segment and the UI is in a partial focus mode where a portion of the data line499is highlighted differently. The UI responds by increasing or maintaining a highlight level of the third intermediate group513of the data line with a lower highlight level for the second segment area512and a lower highlight level for the first segment511. InFIG. 50Cthe UI is in a focus mode when the gesture of the control point506has extended to the touch circle507and the UI responds by selecting and highlighting or otherwise differentiating only the last third segment503of data.

As shown inFIGS. 50A-50C, the user has selected the third intermediate group513, but suppose this was a user error. The user has two options: use the back mechanic to return to the prior UI control system with the entire line499highlighted as shown inFIG. 50A, or alternatively, the user can use the scroll mechanic to move past the artificial boundary created by the intermediate grouping.FIGS. 51A-51Dillustrate a process for using a scroll mechanic to correct the error which can have similar functionality. The user has erroneously selected the third intermediate group513which can be highlighted. InFIG. 51A, and the user initiates a gesture on the UI input to set up a control point521and a touch circle is formed with six sector areas525. The third intermediate group513of the line499is uniformly highlighted so the UI is in a no focus mode. InFIG. 51B, the user has continued the gesture, moving the control point522up to the first segment area525and the UI responds by starting to highlight the first letter in the third intermediate group513while the adjacent letters have lower levels of highlighting. The UI is in a partial focus mode. InFIG. 51C, the user has swiped the control point523to the touch circle and in this embodiment, only the first block of the third intermediate group513is now highlighted and the UI is in a focus mode. With reference toFIG. 51D, the user can use the scroll mechanic and swipe counterclockwise to move over by a left half disc at a time as in the half disc scroll mechanic to move the control point524to the left side of the touch circle whichFIG. 51Dillustrates with segments on the left side of the touch circle. The highlighting is moved from third intermediate group513to the second intermediate group512. The UI can highlight the individual character on the right side of the second intermediate group512, and the UI can be in the focus mode again. The user can release at the desired character to complete the selection gesture.

A menu system can be created based on a 2D-1D primary mechanic in combination with a back mechanic. Menu entries can be laid out either horizontally or vertically in the available space. Nested previews of submenus may be displayed, and multiple gestures can be performed concurrently. Entry text can be fixed or rotated to make more space for previews.

With reference toFIG. 67A-67D, an example of a UI is illustrated with both touch inputs and the visual display. In the illustrated embodiment shown inFIG. 67A, the visual display can include a listing of primary listings671which are in spaces a-h indicated on the left side and for each of the listings there are multiple sub-listings673indicated in a contrasting color or colors such as red. The UI has a 2D input, here described as a touchpad for simplicity, and the user can use the touchpad to set an initial control point675. InFIG. 67B, the user swiped or dragged to the right outward to the scroll selector which includes spaces for a-h. In this example, the user has moved the control point677to the “c” area of the scroll selector. The display can respond by enlarging or increasing the degree of focus on the “c” listing which is “cities” in this example. Because the cities listing is enlarged, the associated sub-listings are also enlarged which include the cities: New Orleans, Paris, Montreal, and Tokyo. The user can continue moving their finger or release the finger from the touch screen which can result in the UI system to cause the sublistings becoming larger and the other text becoming smaller as shown inFIG. 67Cand possibly remove all information other than the sublisting data679which includes the cities: New Orleans, Paris, Montreal, and Tokyo in an animation process on the display of the computing device. With reference toFIG. 67D, the UI can continue to display just the sublisting data679cities New Orleans, Paris, Montreal, and Tokyo in a vertical alignment of blocks. The user can then select a city for an input. This described process can be repeated for further navigation through the sublistings and possibly sub-sublistings.

With reference toFIGS. 68A-68D, another example of a UI for selecting a listing item is illustrated. The navigation through the primary listings, sublistings, and further sublistings are illustrated on the display. With reference toFIG. 68A, the user can use the UI by initially touching the touchpad to set an initial control point681on the touch input area of the computing device and the UI form a touch circle685. On the display portion of the computing device, a listing of topics691which are visible and legible and sublistings693shown as a set of illegible blocks to the right of the topics691.

With reference toFIG. 68B, the user swipes the control point682down. The UI can respond by making the lower listing which in this example is “colors” larger and also enlarging the associated sublisting693. In this example, the listing “colors” is enlarged and the sublisting colors purple, red, blue and green. The user can decide then to change the listing of interest and move the touch swipe or drag in a counterclockwise rotation which results in the focus or enlargement moving upward through the primary listings. With reference toFIG. 68C, the user has swiped in a counterclockwise direction to the control point683which causes the displayed listing691to scroll upward to the topic “animals” enlarged and the sublisting693is now enlarged and visible: dog, cat, rabbit, whale, turtle, bear, and frog are illustrated. With reference toFIG. 68D, the user can continue to rotate the swipe or drag in a counterclockwise direction to the control point684. The UI can respond by changing the primary listing to move upward to “cities” and the sub-listings: New Orleans, Paris, Montreal, and Tokyo are enlarged and visibly displayed. The user can release to select “cities” and the process can be repeated to allow the user to select one of the sublistings to move forward or swipe back to go back to earlier menus.

With reference toFIGS. 69A-69D, the UI can use a “back mechanic” to go back to prior menus during navigation. InFIG. 69A, the user has touched the touch input of the UI to set a control point694and the UI responds by forming a touch circle685. The display in this example computing device shows an animal sublistings698at full size which includes: dog, cat, rabbit, whale, turtle, bear, and frog. With reference toFIG. 69B, the user has dragged or swiped the control point695to the left which results in the display showing the primary listing699which includes: fruits, vehicles, cities, states, animals, countries, months, and colors. The sublisting698for the listing “animals” is shown and legible on the visual display. Additional sublistings associated with the listings above and below the “animals” listing can be shown in a smaller less focused format. With reference toFIG. 69C, the user continues to drag or swipe the control point696to the left which results in smaller less focused sublisting for animals and associated sublistings. InFIG. 69D, the user continues to swipe or drag left to the touch circle685so that all of the primary listings699are displayed at equal sized text and the sublistings698are displayed in equal size in a much smaller text size which may not be legible. The user can then perform additional UI interactions for further navigation through the primary listings699.

With reference toFIGS. 70A-70Eexamples of scrolling and a corresponding visual display are illustrated. As discussed, a listing can have a visible portion701that is shown on the display of the computing device and non-visible portion(s)703that is not displayed on the display of the computing device. The non-visible portion(s)703can be above and/or below the visible portion701and can be moved into the visible portion703through scrolling through the UI. InFIG. 70A, the user has touched the touch pad at the control point704, and the UI forms a touch semi-circle, and then the user swipes right to the control point705. The listings d-f are in the visible portion701with the listing “e” having the largest size or primary focus and the adjacent letters “d” and “f” being smaller in size and the next adjacent letters a-c and g-k are compressed vertically very small and can be illegible. The remaining letters1-zare in the non-visible portion703virtually below the visible portion701and not shown on the display of the computing device.

With reference toFIG. 70B, the user has swiped clockwise to control point706which is adjacent to the vertical line through the touch semi-circle. The swipe to control point706is interpreted by the UI and causes the enlarged visible portion of701to move down to the letters h-k with the listing k having the largest text or primary focus while the upper letters a-g are compressed, much smaller, and possibly illegible. With reference toFIG. 70C, the user can continue to swipe clockwise and the UI can rotate the touch semi-circle counterclockwise about 90 degrees in this example. The visible listings are now m-p with the letter p being the largest or primary focus listing letter and the letters m, n, and o being smaller letters but visible. The letters f-l are above the letter m and compressed and may not be visible.

With reference toFIG. 70Dthe user can continue to scroll clockwise to control point708and the display can move to the display showing letters C-F. The focus of the visible portion701is F, the adjacent letters C, D, and E that can be smaller in size while the letters u-z and A-B can be compressed and not visible. InFIG. 70E, the user can stop and swipe counterclockwise to move back to control point709so that focus of the visible portion701is now B which is the largest size on the visual display, the adjacent letters A and C are in smaller text and the remaining letters v-z and D-F are compressed and may not be visible.

In some embodiments the inventive UI can be used for interacting with something that requires a visual representation like a drawing program or audio editing. The may present the user with a static view of data that can be a visual representation711of the data documents (drawing, wav file, etc.) illustrated on the display of the computing device710as shown inFIG. 71A. By selecting the area of focus and visual representation711with the UI mechanism, the system interpolates and transitions to a dynamic view shown inFIG. 71Bwhich can be one of a series of images712of impulse curve data. For the interpolation, additional images can be displayed with the static image shown inFIG. 71Ashown at the top and additional views712shown underneath in this embodiment.

From the sequence of images, the UI system can be used to zoom and select an image. With reference toFIG. 71C, the user can touch the UI at an initial control point and the UI can create a control circle717. The user can then manipulate the displayed data by swiping on the touchscreen and zooming and/or scrolling. In this example, the control point715swipes outward towards the touch circle717which results in zooming. The zoom level can correspond to the spacing of points719on the touch circle717. At the initial touch circle, the points can be evenly spaced so that the images are all equal in zoom level and size. As discussed, the points719are redistributed to be non-evenly spaced around the touch circle717when the user swipes outward radially. Scrolling can then be performed with angular motion of the control point715as described above so a user can scroll through the images712which results in a flipbook-like fashion display of the images. And, as with all other visualizations, there is no need to scroll linearly.

Again, radial swiping motion results in one of the images being enlarged and other images being reduced in size and rotational swiping results in scrolling. The user can select an image712by swiping to the touch circle. In this example, the dynamic view shown inFIG. 71Cis a normal menu, but in other embodiments, the dynamic view could also work with a satellite circles style menu as well. Satellite circle visualizations can be used to make visual adjustments in a flipbook-like fashion.

Tactile Feedback

In an embodiment, the inventive UI can include a tactile feedback mechanism that can allow users to augment the information provided by visualizations. For example, vibration when the user enters the top or bottom of the input disc could help the user to orient their input when not looking at the screen, in a similar fashion to raised bumps that are placed on the ‘f’ and ‘j’ keys on many key English keyboards. Vibration patterns can also be used to notify the user as they enter a new selection area on the touch circle UI, when a selection becomes focused, etc. More specifically, the tactile feedback can be applied to the touchpad, joystick, or other device which the user can be in contact with. Different vibrations can be used to identify movement of the control point into different regions of the UI control areas. In other embodiments, the UI can emit audio signals in response to help the user identify movement into a UI control point in different regions of the UI control areas.

Previews and Nesting

In some embodiments, the inventive UI can have nested interactions that a user can interact with in an interleaving action. For example, in a two finger embodiment, a user can use one finger to manipulate a folder while the other finger's location is used to manipulate the contents of the subfolders. With reference toFIGS. 65 and 66, examples of nested UI interfaces are illustrated. The primary UI menu651can be controlled by a first finger and a nested menu653can be controlled by a second finger. The initial contact of either finger can result in a control point and radial movement can result in zooming and resizing of the menu components while radial swiping can result in scrolling of the zooming feature through the menu listings.FIG. 65illustrates a radial main menu651and a radial nested menu653having satellite circles. Each of these menus can be independently controlled in any of manners described. In other embodiments, different types of menus can be used. For example,FIG. 66illustrates a radial primary menu651controlled by a first finger and the nested menu653is a rectangular configuration which can be scrolled through as described above.

Use Cases

File Browser

The following is a description of a file browser using the UI with the inventive Scroll Mechanic. As illustrated inFIGS. 52 and 53, a mobile computing device is illustrated with a touchscreen such as a smartphone. The screen can have an upper display portion533and a lower UI control area531unit circle. The system UI can be used as described above and the UI allows users to traverse and edit the project's directory structure, and provides scalable previews into both files and directories. The upper portion533of a touchscreen can display a directory and the size of the directory listings can be configured with a center listing having the largest format and listings having smaller formats the farther they are away from the center listing. InFIG. 52, the user has used the lower UI control area531to cause the UI to focus on the listing “main.cpp”535and some code associated with the main.cpp listing535is displayed to the right. The user can select the main.cpp listing535by swiping outward to the touch circle and the UI can display a subfolder of listings537as shown inFIG. 53. The user has used the UI to navigate to the subfolder file which is enlarged text at the center of the display and subfolder listing are shown to the right. The UI can be used to scroll through the subfolders or to go back. In this example, the subfolder of listings537are shown as: here, are, some, files, they, live, here, yay. The user can use the UI to select the desired subfolder listing537.

Browser Menu

In an embodiment, the UI can be used for browsing file menus or context menus. The menu can be modal. As an example, the starting UI menu can include the options: select and new. If the select option is chosen, the file browser UI enters a select mode that allows the user to select a desired file to review and/or edit. The UI can cause the next file or directory that is selected to become active. File previews can still show during the UI selection process. Multiple files/directories can be active at the same time, and those files/directories can each be highlighted in a different color so that the user can more easily distinguish the different files/directories. After the user completes a selection, the menu can allow the user to cancel the selection via a back mechanic so that a selection error can be corrected. Going backward returns the UI to the prior UI menu and the described selection process can be repeated until the user selection has been corrected.

After at least one file/directory is selected, the UI can provide the user with a menu that can contain one or more of the following options: cancel, move here, delete, copy here, copy and rename, move, and/or rename. The move here can be selected to move the data if the directory has changed. The delete can be selected to delete the data. The copy here can be selected to copy the data to a different file if the directory has changed. The copy and rename can be used to create a copy of the data that is renamed. The move and rename can be used to move and rename the selected data if one data file is selected. After the selected function is performed, the UI menu can return to its start state and the described UI data processing can be repeated.

Rotary Keyboard

In an embodiment, the UI can be used as a keyboard. In different keyboard embodiments, the UI interaction can work with either one or two fingers. Each finger can interact with the touchscreen using a satellite circle visualization.

In a two-finger mode, the touchpad screen can be divided into two columns of equal width, and each column has an associated keyboard menu that is manipulated by beginning a radial gesture in that column by a left finger and a right finger. With reference toFIGS. 59 and 60, a visual display can be displayed above the touchpad keyboard input area visual content595in a visual display area. The keyboard menus start in a grid mode that mimics the QWERTY keyboard. The left half591of the QWERTY keyboard for the left finger, and the right half of the QWERTY keyboard593for the right finger on the lower portion of the touchscreen597of the computing device input. A keyboard “touch area” can be at the bottom of the screen597so that touch UI interactions will not obscure visual content595. As described above, a satellite circle visualization begins with a radial gesture that starts in the touch area. With reference toFIG. 59, the user can touch the left block of letters591and the right block of letters593with a left thumb and a right thumb respectively. With reference toFIG. 60, when a finger touches the screen, the associated menu animates to a satellite circle visualization. The UI has responded to the left thumb and right thumb control points and the letters can be converted into satellite circle visualizations592,594which are illustrated above the touch circles. The UI can function as described above with reference toFIGS. 33-35. In an embodiment, the user can touch the touchscreen to set a control point and swipe outward to cause the visual display to zoom on one of the letters. The user can swipe to the touch circle to select letters. The user can manipulate the UI to select and input data. In this example, the text input cursor599is at the end of the existing text std::cout<<″|.

With reference toFIGS. 61 and 62, another two finger keyboard embodiment is illustrated with the computing device display in a landscape orientation. In this embodiment, the two blocks of letter menus591,593are on opposite sides of the visual content595. This keyboard can function in the same manner described above with reference toFIGS. 59 and 60. When the user touches the letter menus591,593, the UI can respond by displaying satellite circle visualizations592,594which are illustrated above the touch circles.

With reference toFIGS. 63 and 64, an embodiment of a single finger keyboard UI is illustrated. The user can scroll and zoom to any desired letter and then swipe out to the control circle to select the desired letter and the process can be repeated to input text.

One finger mode is similar to two-finger, but there is only one menu at a time. This UI menu will typically contain a combination of the contents of the two corresponding two-finger menus. Some selections (e.g., “punctuation” or “shift”) can change the content of the menu. If the user changes from one side to the other (e.g., when switching hands), the visualization animates to the other side.FIGS. 63 and 64illustrate an example of single finger UIs.FIG. 63illustrates a touchscreen with a lower portion showing a single QWERTY keyboard596and an upper portion which is a visual display595where the user can add text to. The user can touch the touchscreen input596and the UI can respond by displaying a satellite circle visualization598which includes all of the QWERTY keyboard letters above the touch circle. With reference toFIG. 64, the user has touched the touchpad and the letters of the keyboard have been transformed as illustrated above in the satellite circle visualization598which will have all of the letters in the keyboard. The user can input text by selecting specific letters or other typed characters as described above.

For both the single and two finger keyboard inputs, some selections (e.g., “punctuation” or “shift”) can change the contents of the menus. If such a selection is made by one finger, both menus immediately switch to the new static visualization (grid). Any previous interaction with the other finger is canceled.

Another keyboard embodiment is based entirely on a more standard menu system as discussed earlier with reference toFIGS. 67A-69D. Thus, typing can be mixed with arbitrary menu-based operations, such as caps-lock, completion, grouping of characters, etc. As the keyboard is just a menu, this also allows for both multitouch interaction and an extremely user-configurable typing experience. So, for example, a simple user-writable text file could define a graph structure representing a keyboard.

Cursor Placement

The UI can allow a user to place a cursor at any location on a document displayed on the computing device. The UI mechanism can utilize a vertical and horizontal control system to place the cursor at the desired location. In a single finger mode, the single finger can first perform a vertical focus and then a horizontal focus or a horizontal focus and then a vertical focus to place a cursor by the single finger. In a two finger mode, the horizontal focus and the vertical focus can be performed simultaneously by left and right fingers. One embodiment of cursor placement has been discussed previously as color zoom.

With reference toFIGS. 56A-56G, in a one-finger mode, the user can choose a row and column with two independent one-finger gestures, possibly interleaved. With reference toFIGS. 56A-56C, the UI can be used as described above and a first gesture can choose a row. InFIG. 56Athe user has touched the touchpad at the touchpoint in a neutral zone. InFIG. 56B, the user has swiped outward to the upper right to the inner edge of the scroll selector zone. The upper rows of the displayed text are slightly highlighted. InFIG. 56C, the fourth row is more prominently highlighted and the user can release the finger from the touchpad to select the fourth row. If any other row is desired the user can control the UI to scroll to highlight the target row. The user can then use the UI to choose a column for selecting a specific word in the fourth row. With reference toFIG. 56D, the user has touched the touchpad at the touchpoint in a neutral zone. InFIG. 56E, the user has swiped outward to the lower right to the inner edge of the scroll selector zone. The second word in the fourth row of the displayed text is slightly highlighted. InFIG. 56F, the letter “l” in the word “elit” is more prominently highlighted and the user can release the finger from the touchpad to select the letter “l” in the word elit as shown inFIG. 56G.

With reference toFIGS. 57A-57D, a two finger UI input mode can be used to perform the selection of a specific letter in a displayed text on a visual display. The left finger menu can navigate a row selection, and the right finger menu can navigate a column selection. When the user selects both a row and a column, keyboard mode can be launched, with the cursor at the selected row and column. If the user selects just a row or just a column, that selection is “locked in”. A locked in selection can be overridden by starting a new gesture with the associated finger. A gesture that does not make selection locks in the existing locked-in selection if there is one, or it locks in the user's existing position.

FIGS. 57A-57Dillustrate a two-finger mode for performing the same cursor placement function described above with reference toFIGS. 56A-56G. InFIG. 57A, the left hand finger and the right hand finger have touched the touchscreen at two touch control points in two neutral zones. InFIG. 57B, the left finger swipes to the upper right and simultaneously the right finger swipes to the lower right to the inner edge of the inner edge of the scroll selector zone. The text is highlighted in the area around the second word of the fourth row. InFIG. 57C, the letter “l” in the word “elit” is more prominently highlighted and the user can release both fingers from the touchpad to place a cursor between the letters e and1in the word “elit” as shown inFIG. 57D.

If the user begins either finger swipe gesture but fails to make a UI input selection (including back), the user's existing position is selected and the UI can move to the next step. The user can back out of a UI row selection by making a back gesture swipe selection during the column gesture. Note that the user can effectively move within the same row by tapping and then making a column gesture. Likewise, the user can effectively move within the same column by making a row gesture and then tapping.

Choosing One or Two Finger Mode

The UI control mode of the computing device can support one-finger input, and some can support two-finger input. For modes that support both one and two-finger input, the finger mode is chosen as follows. When the user begins a gesture with one finger, one-finger mode is invoked. If at any subsequent point, the user begins a gesture with a second finger, two-finger mode is invoked. If the user doesn't interact with two fingers during an implementation-defined timeout, the mode returns to one-finger.

Local Cursor Navigation

With reference toFIGS. 58A-58C, an example of the inventive UI used for Local Cursor Navigation is illustrated.FIG. 58Aillustrates a block of text displayed on a visual display with a portion of the text highlighted by a rectangular marking such as color or outline.FIG. 58B, the UI allows the user to navigate with a row focus to more narrowly focus around the word “labore”. InFIG. 58Cthe column focus can be applied within a smaller grid surrounding the current cursor location around the letter “o” in the word “labore”. This grid scrolls left, right, up, and down using the normal mechanics. (i.e., when focus gets to the edge of the grid, the grid window moves in that direction using the Scroll Mechanic.) In other embodiments, the vertical row focus and column horizontal focus can be performed by a two finger embodiment of the UI invention. The UI can be used to focus on a single letter in a block of text by applying a row focus using the left finger and simultaneously a column focus can be performed with the other finger. The focusing on the text can be controlled and illustrated to a user on a screen in a highlighted format.

Hardware

FIG. 73shows an example of a generic computer device900and a generic mobile computer device950, which may be used to implement the processes described herein, including the mobile-side and server-side processes for installing a computer program from a mobile device to a computer. Computing device900is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device950is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

Computing device900includes a processor902, memory904, a storage device906, a high-speed interface908connecting to memory904and high-speed expansion ports910, and a low speed interface912connecting to low speed bus914and storage device906. Each of the components processor902, memory904, storage device906, high-speed interface908, high-speed expansion ports910, and low speed interface912are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor902can process instructions for execution within the computing device900, including instructions stored in the memory904or on the storage device906to display graphical information for a GUI on an external input/output device, such as display916coupled to high speed interface908. In other implementations, multiple processors and/or multiple busses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices900may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory904stores information within the computing device900. In one implementation, the memory904is a volatile memory unit or units. In another implementation, the memory904is a non-volatile memory unit or units. The memory904may also be another form of computer-readable medium, such as a magnetic or optical disk.

The high speed controller908manages bandwidth-intensive operations for the computing device900, while the low speed controller912manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In one implementation, the high-speed controller908is coupled to memory904, display916(e.g., through a graphics processor or accelerator), and to high-speed expansion ports910, which may accept various expansion cards (not shown). In the implementation, low-speed controller912is coupled to storage device906and low-speed expansion port914. The low-speed expansion port914, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard936in communication with a computer932, a pointing device935, a scanner931, or a networking device933such as a switch or router, e.g., through a network adapter.

The computing device900may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server920, or multiple times in a group of such servers. It may also be implemented as part of a rack server system924. In addition, it may be implemented in a personal computer such as a laptop computer922. Alternatively, components from computing device900may be combined with other components in a mobile device (not shown), such as device950. Each of such devices may contain one or more of computing devices900,950, and an entire system may be made up of multiple computing devices900,950communicating with each other.

Computing device950includes a processor952, memory964, an input/output device such as a display954, a communication interface966, and a transceiver968, among other components. The device950may also be provided with a storage device, such as a Microdrive, solid state memory or other device, to provide additional storage. Each of the components computing device950, processor952, memory964, display954, communication interface966, and transceiver968are interconnected using various busses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

The processor952can execute instructions within the computing device950, including instructions stored in the memory964. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor may provide, for example, for coordination of the other components of the device950, such as control of user interfaces, applications run by device950, and wireless communication by device950.

Processor952may communicate with a user through control interface958and display interface956coupled to a display954. The display954may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface956may comprise appropriate circuitry for driving the display954to present graphical and other information to a user. The control interface958may receive commands from a user and convert them for submission to the processor952. In addition, an external interface962may be provided in communication with processor952, so as to enable near area communication of device950with other devices. External interface962may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

The memory964stores information within the computing device950. The memory964can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory974may also be provided and connected to device950through expansion interface972, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory974may provide extra storage space for device950, or may also store applications or other information for device950. Specifically, expansion memory974may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory974may be provided as a security module for device950, and may be programmed with instructions that permit secure use of device950. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

The computing device950may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone980. It may also be implemented as part of a smartphone982, personal digital assistant, a tablet computer983or other similar mobile computing device.

It should be appreciated that the described embodiments can be implemented in numerous ways, including as a process, an apparatus, a system, a device, a method, or a computer-readable medium such as a computer-readable storage medium containing computer-readable instructions or computer program code, or as a computer program product, comprising a computer-usable medium having a computer-readable program code embodied therein.

Aspects of the methods, processes, and systems described herein may be implemented as functionality programmed into any of a variety of circuitry, including programmable logic devices (“PLDs”), such as field programmable gate arrays (“FPGAs”), programmable array logic (“PAL”) devices, electrically programmable logic and memory devices and standard cell-based devices, as well as application specific integrated circuits. Method and process steps may be executed by server or client side components that are processor-based components, programmed digital circuits, programmed arrays, and so on. Method steps may also be embodied as executable program code executed in a processor based system, such as a server computer or client computer coupled in a network. The terms, “component,” “module,” “procedure,” “unit,” may all refer to a circuit that executes program code to perform a function implemented in software.

It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and/or as data and/or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and/or other characteristics. Computer-readable media in which such formatted data and/or instructions may be embodied include, but are not limited to, non-volatile storage media in various forms (e.g., optical, magnetic or semiconductor storage media).

Embodiments are directed to a processing component in or associated with a server computer used by a software developer to implement an application-specific action and/or persistence interface, develop a small configuration file that directs the workflow engine, and provide a file that contains the formal API definition. This is a data-centric model that frees developers from implementing specific and customized code to handle how to get data to and from an API. Built-in support is provided for authentication, list processing, data paging, data transformations and multi-threading.