Patent Description:
According to the present invention, there is provided a method according to claim <NUM>, a device according to claim <NUM>, and a computer-readable storage medium according to claim <NUM>. In general, this disclosure is directed to techniques for utilizing wearable computing devices to facilitate direct manipulation of objects in a graphical user interface output by a display device that is not presence-sensitive. For instance, as opposed to using a mouse or other indirect manipulation tool, a user may move a wearable computing device around on a display device in order to directly manipulate objects displayed at the display device. One or more sensors of the wearable computing device, such as one or more optical sensors and/or one or more motion sensors, may generate data as the wearable computing device is moved on the display device. A control device (e.g., that is causing the display device to output the graphical user interface) may determine a location of the wearable computing device on the display device based on the sensor data. The control device may then use the determined location of the wearable computing device as user input to manipulate objects included in the graphical user interface.

In this way, the techniques of this disclosure enable a user to directly manipulate objects displayed by a display device. By utilizing the determined location of the wearable computing device as user input, the control device may enable richer interactions with display devices that do not include touchscreens or other direct manipulation means.

As one example, a method includes determining, by one or more processors, an estimated location of a wearable computing device on a display device that is not presence-sensitive, wherein the wearable computing device is configured to be worn on a wrist of a user; causing the display device to output, at the estimated location, a graphical pattern; determining, by the one or more processors and based on optical data generated by an optical sensor of the wearable computing device, a refined location of the wearable computing device; and utilizing, by the one or more processors, the refined location of the wearable computing device as a location of user input for an application outputting a graphical user interface via the display device.

As another example, a control device includes one or more processors; and a memory comprising instructions that, when executed by the one or more processors, cause the one or more processors to determine an estimated location of a wearable computing device on a display device that is not presence-sensitive, wherein the wearable computing device is configured to be worn on a wrist of a user; cause the display device to output, at the estimated location, a graphical pattern; determine, based on optical data generated by an optical sensor of the wearable computing device, a refined location of the wearable computing device; and utilize the refined location of the wearable computing device as a location of user input for an application outputting a graphical user interface via the display device.

As another example, a computer-readable storage medium stores instructions that, when executed, cause one or more processors to determine an estimated location of a wearable computing device on a display device that is not presence-sensitive, wherein the wearable computing device is configured to be worn on a wrist of a user; cause the display device to output, at the estimated location, a graphical pattern; determine, based on optical data generated by an optical sensor of the wearable computing device, a refined location of the wearable computing device; and utilize the refined location of the wearable computing device as a location of user input for an application outputting a graphical user interface via the display device.

<CIT> discloses a method of cursor control with an optical wrist-worn device that detects the position of a cursor on a display screen, relative to a reference frame from reflectors placed on the display bezel.

<CIT> and <CIT> disclose an interactive display system including a wireless pointing device, and positioning circuitry capable of determining absolute and relative positions of the display at which the pointing device is aimed. An error value between the absolute position and an estimated or actual relative position at the point in time of the absolute position is determined, and a compensation factor is determined from this error value that is applied to subsequent relative positioning results.

<FIG> is a system diagram illustrating a system that enables a wearable computing device to be used to facilitate direct manipulation of objects in a graphical user interface output by a display device, in accordance with one or more techniques of this disclosure. As shown in <FIG>, system <NUM> includes display device <NUM>, wearable computing device <NUM>, and control device <NUM>.

Display device <NUM> may be configured to display graphical information for viewing by one or more users. Examples of display device <NUM> include, but are not limited to, monitors, televisions, rear-projections screens, or any other device capable of displaying information. Display device <NUM> may form an output images using any suitable technology. Example image formation technologies include, but are not limited to, liquid crystal display (LCD), laser, light emitting diode (LED), organic light emitting diode (OLED), plasma, and the like.

Display device <NUM> may include components configured to generate the graphical information for display. For instance, display device <NUM> may be a so-called "smart display. " Display device <NUM> may include one or more inputs (e.g., one or both of wired and wireless inputs) capable of receiving graphical information for display from an external device (e.g., control device <NUM>). For instance, display device <NUM> may include one or more wired inputs (e.g., High-Definition Multimedia Interface (HDMI) inputs, one or more display-port inputs, one or more network connections (e.g., WIFI, BLUETOOTH, Ethernet, etc.), and/or one or more digital visual interface (DVI) inputs.

Control device <NUM> may be configured to cause display device <NUM> to output graphical information. For instance, control device <NUM> may cause display device <NUM> to display a graphical user interface that includes one or more graphical objects (e.g., buttons, keys, sliders, and the like). In some examples, control device <NUM> may be integrated into display device <NUM>. For instance, where display device <NUM> is a "smart display" control device <NUM> may be integrated into display device <NUM>. In other examples, control device <NUM> may be an external device to display device <NUM>. For instance, control device <NUM> may output the graphical information to display device <NUM> via an input of display device <NUM>. As one example, control device <NUM> may be a streaming stick plugged into an HDMI input of display device <NUM>. As another example, control device <NUM> may be a mobile computing device (e.g., a smartphone) and may wirelessly stream the graphical information to display device <NUM>.

Control device <NUM> may include one or more processors that may perform various computations to support operation of control device <NUM>. For instance, one or more processors of control device <NUM> may execute an application that generates a graphical user interface, which control device <NUM> may cause to be displayed at display device <NUM>.

In some examples, display device <NUM> may include one or more input components configured to enable a user to directly manipulate graphical objects displayed at display device <NUM>. For instance, display device <NUM> may include a touch-sensitive or other presence-sensitive panel that detects user movements on or near display device <NUM>. Where display device <NUM> includes a presence-sensitive panel, display device <NUM> may output representations of user input received via the presence-sensitive panel to control device <NUM>. For instance, display device <NUM> may output a stream of (x,y) coordinate pairs to control device <NUM>, the coordinate pairs representing locations on display device <NUM> at-which the presence-sensitive panel detected user input. Control device <NUM> may utilize the representations of user input provided by display device <NUM> as user input for an application executing at control device <NUM> that is outputting a graphical user interface being displayed at display device <NUM>.

As discussed above, direct manipulation of displayed graphical objects may be desirable over indirect manipulation. For example, direct manipulation of graphical objects may feel more natural to the user than indirect manipulation. However, in some examples, display device <NUM> may not include any input components configured to enable a user to directly manipulate graphical objects displayed at display device <NUM>. For instance, display device <NUM> may not be a so-called "touchscreen" and may not be considered a presence-sensitive display. As such, it may be desirable to enable direct manipulation of graphical objects displayed at display device <NUM> even when display device <NUM> is not a presence-sensitive display.

In accordance with one or more techniques of this disclosure, control device <NUM> may enable a user to use wearable computing device <NUM> to perform direct manipulation of graphical objects displayed at display device <NUM>. Wearable computing device <NUM>, (referred to simply as "wearable <NUM>") may be a device configured to be worn by a user. In the example of <FIG>, wearable <NUM> is a computerized watch. However in other examples, wearable <NUM> is a computerized fitness band/tracker, computerized eyewear, computerized headwear, a computerized glove, etc. As discussed below, in some examples, a device other than a wearable device may perform the functions of wearable <NUM>.

Wearable <NUM> includes sensors <NUM> that may be configured to generate data representing one or more measured parameters. As discussed in further detail with reference to <FIG>, sensors <NUM> may include one or more motion sensors and one or more optical sensors. Wearable <NUM> may include components configured to wirelessly transmit data generated by sensors <NUM> to one or more external devices, such as control device <NUM>.

Control device <NUM> may determine a location of wearable <NUM> on display device <NUM> based on the sensor data received from wearable <NUM>. For instance, control device <NUM> may determine an estimated location of wearable <NUM> on display device <NUM> (e.g., based on motion data generated by one or more motion sensors of sensors <NUM>). As a user moves wearable <NUM> across display device <NUM>, the motion sensors of wearable <NUM> may generate motion data that represents the movement of wearable <NUM>. Control device <NUM> may process this motion data to determine a relative movement of wearable <NUM>. For instance, based on accelerometer and/or gyroscopic data generated by the motion sensors, control device <NUM> may determine that wearable <NUM> has moved <NUM> inches with a heading of <NUM>°. However, determining the location of wearable <NUM> on display device <NUM> based solely on motion data may present one or more disadvantages. For instance, even with an initially known location, control device <NUM> may use dead reckoning to determine subsequent locations, which may introduce more and more error over time. As such, in some examples, it may be desirable to determine the location of wearable <NUM> on display device <NUM> based on more than just motion data.

In some examples, in addition to or in place of motion data, control device <NUM> may determine the location of wearable <NUM> on display device <NUM> based on optical data generated by one or more optical sensors of wearable <NUM>. For instance, control device <NUM> may cause display device <NUM> to output a graphical pattern at an estimated current location of wearable <NUM> on display device <NUM>. As shown in <FIG>, control device <NUM> may cause display device <NUM> to output graphical pattern <NUM>, which includes a plurality of regions that each have a different optical characteristic (e.g., a different color, a different shade of color or brightness). The plurality of regions may include center region <NUM> and a plurality of border regions 16A-16D (collectively, "border regions <NUM>") dispersed about center region <NUM>. Control device <NUM> may use optical data received from wearable <NUM> to refine the estimate of the current location of wearable <NUM> on display device <NUM>. Control device <NUM> may utilize the refined location as user input for an application executing at control device <NUM> that is outputting a graphical user interface being displayed at display device <NUM>. Further details of one example technique for refining an estimated location of wearable <NUM> are discussed below with reference to <FIG>.

In some examples, control device <NUM> may determine the location of wearable <NUM> on display device <NUM> based on a combination of motion data and optical data. For instance, control device <NUM> may determine the initial estimated location of wearable <NUM> based on motion data and determine the refined location based on optical data.

Control device <NUM> may utilize the refined location of the wearable computing device as a location of user input for an application outputting a graphical user interface via display device <NUM>. For instance, where an application executing at control device <NUM> outputs a graphical user interface including a graphical button at location x1,y1 on display device <NUM>, the application may determine that the graphical button has been selected where the refined location is at x1,y1.

While illustrated in <FIG> as being a separate device, control device <NUM> may be a separate device, or may be integrated into display device <NUM> or wearable <NUM>. As one example, where control device <NUM> is a separate device (i.e., that is different than display device <NUM> and wearable <NUM>), control device <NUM> may be a laptop or smartphone that receives (e.g., wirelessly) sensor data from wearable <NUM>, executes an application to generate a graphical user interface, causes display device <NUM> to display the graphical user interface, and generates user input for the application based on the received sensor data. As another example, where control device <NUM> is included in display device <NUM>, the functionality described herein may be performed by one or more processors of display device <NUM>. For instance, one or more processors of display device <NUM> may receive sensor data from wearable <NUM>, execute an application to generate a graphical user interface, display (at display device <NUM>) the graphical user interface, and generate user input for the application based on the received sensor data. As another example, where control device <NUM> is included in wearable <NUM>, the functionality described herein may be performed by one or more processors of wearable <NUM>. For instance, one or more processors of wearable <NUM> may obtain sensor data generated by sensors <NUM> of wearable <NUM>, execute an application to generate a graphical user interface, cause display device <NUM> to display the graphical user interface (e.g., wirelessly transmit instructions to generate the user interface to display device <NUM>), and generate user input for the application based on the received sensor data.

<FIG> is a block diagram illustrating wearable computing device <NUM> (referred to simply as "wearable <NUM>") that may be used to facilitate direct manipulation of displayed user interface components, in accordance with one or more aspects of the present disclosure. In the example of <FIG>, wearable <NUM> is a computerized watch. However in other examples, wearable <NUM> is a computerized fitness band/tracker, computerized eyewear, computerized headwear, a computerized glove, etc..

As shown in <FIG>, in some examples, wearable <NUM> may include attachment component <NUM> and housing <NUM>. Housing <NUM> of wearable <NUM> includes a physical portion of wearable <NUM> that houses a combination of hardware, software, firmware, and/or other electrical components of wearable <NUM>. For example, <FIG> shows that within housing <NUM>, wearable <NUM> may include sensors <NUM> (including motion sensors <NUM> and optical sensors <NUM>), communication units <NUM>, and communication module <NUM>.

Attachment component <NUM> may include a physical portion of a wearable computing device that comes in contact with a body (e.g., tissue, muscle, skin, hair, clothing, etc.) of a user when the user is wearing wearable <NUM> (though, in some examples, portions of housing <NUM> may also come in contact with the body of the user). For example, in cases where wearable <NUM> is a watch, attachment component <NUM> may be a watch band that fits around a user's wrist and comes in contact with the skin of the user. In examples where wearable <NUM> is eyewear or headwear, attachment component <NUM> may be a portion of the frame of the eyewear or headwear that fits around a user's head, and when wearable <NUM> is a glove, attachment component <NUM> may be the material of the glove that conforms to the fingers and hand of the user. In some examples, wearable <NUM> can be grasped and held from housing <NUM> and/or attachment component <NUM>.

Motion sensors <NUM> represent one or more motion sensors or input devices configured to detect indications of movement (e.g., data representing movement) associated with wearable <NUM>. Examples of motion sensors <NUM> include accelerometers, speed sensors, gyroscopes, magnetometers, tilt sensors, barometers, proximity sensors, or any and all other types of input devices or sensors that can generate data from which wearable <NUM> can determine movement. In some examples, motion sensors <NUM> may be, or may be included in, an inertial measurement unit (IMU). Motions sensors <NUM> may generate "raw" motion data when housing <NUM> moves. The motion data may indicate one or more characteristics of movement including at least one of an acceleration, a distance traveled, a direction, a speed, a degree of rotation, or a degree of orientation.

Optical sensors <NUM> represent one or more devices configured to generate an electrical signal representative of light intensity. Examples of optical sensors <NUM> include, photodiodes, photoresistors, or any other device capable of transducing light into electrical signals. In some examples, one or more of optical sensors <NUM> may be included in a photoplethysmogram (PPG) sensor (e.g., a sensor that is used for heart rate monitoring). Optical sensors of optical sensors <NUM> that are included in the PPG sensor may be disposed on a surface of wearable <NUM> that is against a skin of a wearer of wearable <NUM> when wearable <NUM> is worn. For instance, where wearable <NUM> is a watch, optical sensors of optical sensors <NUM> that are included in the PPG sensor may be located on a bottom of the watch. In some examples, optical sensors <NUM> may be located somewhere on wearable <NUM> other than the surface of wearable <NUM> that is against the skin of the wearer of wearable <NUM> when wearable <NUM> is worn. For instance, optical sensors <NUM> may be included in a front facing camera (or other front facing (e.g., on the watch face) optical sensor) of wearable <NUM>. In such examples, the user may utilize wearable <NUM> to perform direct manipulation without having to remove wearable <NUM> from their wrist.

Communication units <NUM> may communicate with external devices via one or more wired and/or wireless networks by transmitting and/or receiving network signals on the one or more networks. Examples of communication units <NUM> include a network interface card (e.g. such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and/or receive information. Other examples of communication units <NUM> may include short wave radios, cellular data radios, wireless network radios (e.g., WIFI radios, BLUETOOTH radios, etc.), as well as universal serial bus (USB) controllers.

In some examples, wearable <NUM> may include one or more input components that are configured to receive input. Examples of input are tactile, audio, kinetic, presence, and optical input, to name only a few examples. The input components include, in one example, a mouse, keyboard, voice responsive system, video camera, buttons, control pad, microphone or any other type of device for detecting input from a human or machine. In some examples, the input components may include a presence-sensitive input component, which may include a presence-sensitive screen, touch-sensitive screen, radar, etc..

Communication module <NUM> may perform various operations to manage communications between wearable <NUM> and external devices. As one example, communication module <NUM> may receive data from sensors <NUM> (e.g., optical data from optical sensors <NUM> and/or motion data from motion sensors <NUM>), and cause one or more of communication units <NUM> to wirelessly transmit a representation of the data to an external device (e.g., control device <NUM> of <FIG> and <FIG>). As another example, where wearable <NUM> performs the functionality described herein as attributed to control device <NUM>, communication module <NUM> may cause one or more of communication units <NUM> to transmit instructions to generate a graphical user interface to display device <NUM>. In this way, in some examples, wearable <NUM> may internally process the data generated by sensors <NUM> to enable direct manipulation of graphical objects via wearable <NUM>. As another example, communication module <NUM> may cause one or more of communication units <NUM> to transmit an indication of user input received via one or more input components of wearable <NUM> to an external device (e.g., control device <NUM> of <FIG> and <FIG>). In this way, wearable <NUM> may enable a user to use the input components of wearable <NUM> to provide further input for interacting with the displayed graphical objects (e.g., enable the user to "click" on graphical objects).

While described as being a wearable computing device, it is understood that other devices may perform the functions ascribed to wearable <NUM>. For instance, any computing device comprising sensors (e.g., one or more motion sensors and one or more optical sensors) and one or more wireless communication units may perform the functions ascribed to wearable <NUM>. Some examples of such devices include smartphones, tablets, and the like.

<FIG> is a block diagram illustrating an example control device configured to enable use of a wearable device for direct manipulation of displayed graphical objects, in accordance with one or more aspects of the present disclosure. Control device <NUM> of <FIG> is described below within the context of system <NUM> of <FIG>. <FIG> illustrates only one particular example of control device <NUM> and many other examples of control device <NUM> may be used in other instances. Control device <NUM> of <FIG> may include a subset of the components included in example control device <NUM> or may include additional components not shown in <FIG>.

As shown in the example of <FIG>, control device <NUM> includes one or more processors <NUM>, one or more output devices <NUM>, one or more input devices <NUM>, one or more communication units <NUM>, and one or more storage devices <NUM>. Storage devices <NUM> may also include user interface (UI) module <NUM>, communication module <NUM>, and application module <NUM>.

Communication channels <NUM> may interconnect each of the components <NUM>, <NUM>, <NUM>, <NUM>, <NUM> for inter-component communications (physically, communicatively, and/or operatively). In some examples, communication channels <NUM> may include a power bus, a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data or transferring power.

One or more output devices <NUM> may generate output. Examples of output are tactile, audio, and video output. Output devices <NUM>, in one example, includes a presence-sensitive display, sound card, video graphics adapter card, speaker, cathode ray tube (CRT) monitor, liquid crystal display (LCD), or any other type of device for generating output to a human or machine.

One or more input devices <NUM> may receive input. Examples of input are tactile, audio, and video input. Input devices <NUM>, in some examples, includes a presence-sensitive display, touch-sensitive screen, mouse, keyboard, voice responsive system, video camera, microphone, sensor, or any other type of device for detecting input from a human or machine.

One or more communication units <NUM> may communicate with external devices via one or more wired and/or wireless networks by transmitting and/or receiving network signals on the one or more networks. Examples of communication units <NUM> include a network interface card (e.g. such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and/or receive information. Other examples of communication units <NUM> may include short wave radios, cellular data radios, wireless network radios (e.g., WIFI radios, BLUETOOTH radios, etc.), as well as universal serial bus (USB) controllers, HDMI controllers, and the like.

Communication module <NUM> may perform various operations to manage communications between control device <NUM> and external devices. As one example, communication module <NUM> may receive data from one or more components of control device <NUM>, and cause one or more of communication units <NUM> to transmit a representation of the data to an external device. For instance, communication module <NUM> may receive instructions to generate a graphical user interface from an application of application modules <NUM> and cause one or more of communication units <NUM> to transmit the instructions to display device <NUM>. As another example, communication module <NUM> may receive data from one or more of communication units <NUM> that has been wirelessly received from an external device. For instance, communication module <NUM> may receive sensor data (e.g., one or both of motion data and optical data) from one or more of communication units <NUM> that has been wirelessly received from wearable <NUM>.

One or more storage devices <NUM> within source control device <NUM> may store information for processing during operation of control device <NUM> that modules <NUM>, <NUM>, and <NUM> access during execution at control device <NUM>). In some examples, storage device <NUM> is a temporary memory, meaning that a primary purpose of storage device <NUM> is not long-term storage. Storage devices <NUM> on control device <NUM> may configured for short-term storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art.

Storage devices <NUM>, in some examples, include one or more computer-readable storage media. Storage devices <NUM> may be configured to store larger amounts of information than volatile memory. Storage devices <NUM> may further be configured for long-term storage of information as non-volatile memory space and retain information after power on/off cycles. Examples of non-volatile memories include magnetic hard discs, optical discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Storage devices <NUM> may store program instructions and/or information (e.g., data) associated with modules <NUM>, <NUM>, and <NUM>.

One or more processors <NUM> may implement functionality and/or execute instructions within control device <NUM>. For example, processors <NUM> may receive and execute instructions stored by storage devices <NUM> that execute the functionality of modules <NUM>, <NUM>, and <NUM>. Processors <NUM> may execute instructions of modules <NUM>, <NUM>, and <NUM> to cause to perform various actions or functions of control device <NUM>.

Application modules <NUM> represent all the various individual applications and services executing at and accessible from control device <NUM> that may utilize user input. A user may interact with a graphical user interface associated with one or more of application modules <NUM> to cause control device <NUM> to perform a function. Numerous examples of application modules <NUM> may exist and include, a fitness application, a calendar application, a personal assistant or prediction engine, a search application, a map or navigation application, a transportation service application (e.g., a bus or train tracking application), a social media application, a game application, a presentation application (e.g., for presenting slides), an e-mail application, a chat or messaging application, an Internet browser application, or any and all other applications that may execute at control device <NUM>.

UI module <NUM> manages user interactions with other components of control device <NUM>. In other words, UI module <NUM> may act as an intermediary between various components of control device <NUM> to make determinations based on user input and generate output in response to the user input. UI module <NUM> may receive instructions from an application, service, platform, or other module of control device <NUM> to cause a device (e.g., display device <NUM> of <FIG>) to output a user interface. For instance, an application of application modules <NUM> may interact with UI module <NUM> to cause display device <NUM> to display a graphical user interface that includes one or more graphical objects. UI module <NUM> may manage inputs received by control device <NUM> as a user views and interacts with the user interface and update the user interface in response to receiving additional instructions from the application, service, platform, or other module of control device <NUM> that is processing the user input.

In accordance with one or more techniques of this disclosure, control device <NUM> may utilize sensor data generated by a wearable computing device in order to enable the wearable computing device to be used to provide direct manipulation of displayed graphical objects. For instance, UI module <NUM> may determine an estimated current location of a wearable device on a display device using motion data generated by a motion sensor of the wearable device. UI module <NUM> may use optical data generated by one or more optical sensors of the wearable device to refine the estimated location. An application module of application modules <NUM> (e.g., that is outputting a graphical user interface via the display device) may utilize the refined location of the wearable computing device as a location of user input. For instance, UI module <NUM> may provide the application module with the x,y coordinate of the refined location, and the application module may use the x,y coordinate as user input.

The application module may agnostic as to whether the techniques of this disclosure were used to generate the user input location. For instance, to the application module, the location of user input may appear the same as if it were generated using a standard touch screen. As such, the application module may not need to be modified in order to operate using user input locations determined using wearable device enabled direct manipulation as described herein.

In some examples, it may be desirable for a user to interact with displayed objects using input in addition to location. For instance, in addition to being able to a wearable computing device (e.g., wearable <NUM>) to provide user input as a location, it may be desirable for the user to be able to provide user input such as clicks, scrolling, selecting, or other such things. In accordance with one or more techniques of this disclosure, control device <NUM> may receive data from the wearable computing device that represents user input received via one or more input components of the wearable computing device. For instance, UI module <NUM> may receive an indication that a user has tapped on a presence-sensitive display of the wearable computing device. UI module <NUM> may provide the application module with the representation of user input. For instance, where UI module <NUM> receives an indication that a user has tapped on a presence-sensitive display of the wearable computing device, UI module <NUM> may provide the application with user input indication that a click has been received at the x,y coordinate of the refined location.

<FIG> are conceptual diagrams illustrating an example location refinement process, in accordance with one or more techniques of this disclosure. The refinement process of <FIG> is described with reference to system <NUM> of <FIG> that includes display device <NUM>, wearable <NUM>, and control device <NUM>.

Control device <NUM> may cause display device <NUM> to display a graphical user interface that includes one or more graphical objects. For instance, processors <NUM> of control device <NUM> may execute an application module of application modules <NUM>. The executed application module may, with UI module <NUM>, generate instructions to render the graphical user interface. Processors <NUM> may execute communication module <NUM> to cause communication units <NUM> to output the instructions to generate the graphical user interface to display device <NUM>. In general, the graphical objects may be any graphical element that a user may interact with in order to modify operation of control device <NUM>. Example graphical objects include, but are not limited to, dials, buttons, toggles, sliders, keys of a graphical keyboard, and the like.

A user may desire to directly manipulate one or more graphical objects included in the graphical user interface displayed at display device <NUM>. However, in examples where display device <NUM> is not a presence-sensitive display, it may not be possible for the user to merely use their fingers to directly manipulate the displayed objects. As such, in accordance with one or more techniques of this disclosure and as discussed above, the user may utilize a wearable computing device to perform the direct manipulation. For instance, where wearable <NUM> is worn on a wrist of the user, the user may remove wearable <NUM> (e.g., undo attachment component <NUM>) and utilize wearable <NUM> to directly manipulate the graphical objects.

Control device <NUM> may perform operations to enable the user to utilize wearable <NUM> to directly manipulate the graphical objects displayed at display device <NUM>. As discussed above, in general, control device <NUM> may utilize motion data generated by motion sensors <NUM> of wearable <NUM> to determine an estimated location on display device <NUM>, causing display device <NUM> to display a graphical pattern at the estimated location, and then refine the estimated location based on optical data generated by optical sensors <NUM> of wearable <NUM>.

Initially, control device <NUM> may not be able to determine an estimated location of wearable <NUM>. For instance, before the user places wearable <NUM> against display device <NUM>, control device <NUM> may not be able to determine a location of wearable <NUM>. As such, for the initial location, control device <NUM> may output the graphical pattern at a location on display device <NUM> with an instruction for the user to place wearable <NUM> at the location of the displayed graphical pattern. Control device <NUM> may always use a particular location on display device <NUM> as the initial location, or may use different locations depending on context. Once the user places wearable <NUM> on display device <NUM> at the location of the displayed graphical pattern, control device <NUM> may begin to refine the determined location of wearable <NUM>.

The graphical pattern output at display device <NUM> may be designed to facilitate detection by optical sensors, such as optical sensors <NUM> of wearable <NUM>. In general, the graphical pattern may include a plurality of regions that each have a different optical characteristic. As shown in <FIG> and <FIG>, graphical pattern <NUM> includes center region <NUM> and a plurality of border regions <NUM> dispersed about center region <NUM>. While illustrated as including four border regions, the graphical pattern may include any number of border regions. For instance, in some examples, the graphical pattern may include <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or more border regions.

Each of the border regions may be associated with an adjustment direction. The adjustment direction of a particular border region may represent a spatial relationship between the border region and the center region. As one example, the adjustment direction of border region 16A may be <NUM>° (e.g., the direction heading from center region <NUM> to border region 16A). Similarly, the adjustment direction of border region 16B may be <NUM>°, the adjustment direction of border region 16C may be <NUM>°, and the adjustment direction of border region 16D may be <NUM>°. As discussed in further detail below, responsive to optical data indicating that a particular border region of border regions <NUM> is displayed under wearable <NUM>, control device <NUM> may adjust the estimated location of wearable <NUM> in the adjustment direction corresponding to the particular border region.

As shown in <FIG>, the user may initially place wearable <NUM> at the location on display device <NUM> at which graphical pattern <NUM> is displayed. While the user is using wearable <NUM> to perform direct manipulation, control device <NUM> may use a location of the displayed graphical pattern as user input. For instance, UI module <NUM> of control device <NUM> may utilize the x,y coordinate of a centroid of center region <NUM> of graphical pattern <NUM> as user input for an application of application modules <NUM> executing at control device <NUM>.

As shown in <FIG>, the user may move wearable <NUM> across display device <NUM>. As wearable <NUM> is moved, motion sensors <NUM> may generate motion data that represents the movement. Location estimation module (LEM) <NUM> of control device <NUM> may determine an updated estimated location of wearable <NUM> based on the motion data. For instance, LEM <NUM> may perform a double integration of acceleration data generated by accelerometers of motion sensors <NUM>, and add the resulting value to the previously known location of wearable <NUM> (i.e., the location at which graphical pattern <NUM> was displayed before wearable <NUM> was moved).

As shown in <FIG>, control device <NUM> may cause display device <NUM> to output graphical pattern <NUM> at the updated estimated location. Depending on the accuracy of the location estimate, graphical pattern <NUM> may or may not be displayed at the actual location of wearable <NUM>. Location refinement module (LRM) <NUM> of control device <NUM> may refine the estimated location based on optical data generated by one or more optical sensors <NUM> of wearable <NUM>.

One of three scenarios is possible during any point during the refinement process. In a first scenario, graphical pattern <NUM> may not be displayed under wearable <NUM> (e.g., optical sensors <NUM> may not be able to detect any region of graphical pattern <NUM>). In a second scenario, one of border regions <NUM> of graphical pattern <NUM> may be displayed under wearable <NUM>. In a third scenario, center region <NUM> of graphical pattern <NUM> may be displayed under wearable <NUM>. LRM <NUM> may determine which scenario is occurring based on optical data generated by one or more optical sensors <NUM>. As one example, where the optical data indicates that no regions of center region <NUM> and border regions <NUM> are detected, LRM <NUM> may determine that the first scenario is occurring. As another example, where the optical data indicates that a particular border region of border regions <NUM> is detected, LRM <NUM> may determine that the second scenario is occurring. As another example, where the optical data indicates that center region <NUM> is detected, LRM <NUM> may determine that the third scenario is occurring.

LRM <NUM> may perform a particular action based on the determined scenario. Responsive to determining that the first scenario is occurring (e.g., that no regions of graphical pattern <NUM> are detected), LRM <NUM> may increase a display size of graphical pattern <NUM>. In other words, responsive to determining that none of the plurality of regions of graphical pattern <NUM> are detected, LRM <NUM> may cause display device <NUM> to output graphical pattern <NUM> at an increased size relative to a current size. By increasing the display size of graphical pattern <NUM>, LRM <NUM> may increase the likelihood that a region of graphical pattern <NUM> will be detected in the optical data. An example of the size increase can be seen in <FIG>.

Responsive to determining that the second scenario is occurring (e.g., that a particular border region of border regions <NUM> is detected), LRM <NUM> may adjust a displayed location of graphical pattern <NUM>. For instance, responsive to determining that a particular border region of border regions <NUM> is detected, LRM <NUM> may determine, based on the particular border region, an adjustment direction. As discussed above, each border region of border regions <NUM> may be associated with a respective adjustment direction that represents a spatial relationship between the respective border region and center region <NUM>. LRM <NUM> may adjust, in the adjustment direction, the estimated location to obtain a refined location. In the example of <FIG> and <FIG>, LRM <NUM> may determine that border region 16B is detected, determine that the adjustment direction corresponding to border region 16B is a heading of <NUM>°, and move a displayed location of graphical pattern <NUM> a distance with a heading of <NUM>°. The distance that LRM <NUM> moves graphical pattern <NUM> may be a function of the current display size of graphical pattern <NUM>. For example, the distance may be one-half of a radius of the current display size of graphical pattern <NUM>.

Responsive to determining that the third scenario is occurring (e.g., that the center region is detected), LRM <NUM> may decrease a display size of graphical pattern <NUM>. In other words, responsive to determining that center region <NUM> of graphical pattern <NUM> is detected, LRM <NUM> may cause display device <NUM> to output graphical pattern <NUM> at a reduced size relative to a current size. By reducing the display size of graphical pattern <NUM>, LRM <NUM> may reduce the amount of the graphical user interface covered up / occluded by graphical pattern <NUM>. In some examples, LRM <NUM> may reduce the size of graphical pattern <NUM> to a point where graphical pattern <NUM> is not visible around wearable <NUM> (e.g., graphical pattern <NUM> may be totally occluded by wearable <NUM>). An example of the size decrease can be seen in <FIG>.

It is noted that control device <NUM> may continue to perform the refinement process until one or more conditions are met and/or until wearable <NUM> is moved again. One example condition includes graphical pattern <NUM> reaching a minimum display size. The minimum display size may be selected based on a size of wearable <NUM> (e.g., such that the minimum display size is the same as, or slightly smaller than, the size of wearable <NUM>).

It is further noted that certain refinement operations may trigger other scenarios. For instance, reducing the display size of graphical pattern <NUM> may result in center region <NUM> no longer being detected in the optical data (e.g., and either a border region is detected or no region is detected). As one specific example, after reducing the display size of graphical pattern <NUM>, control device <NUM> may determine that a border region is now detected and may adjust the display location of graphical pattern <NUM> based on the detected border region. This process may loop until a minimum size of graphical pattern <NUM> is reached.

<FIG> is a flow diagram illustrating example operations of a device to enable use of a wearable device for direct manipulation of displayed graphical objects, in accordance with one or more techniques of the present disclosure. The techniques of <FIG> may be performed by one or more processors of a device, such as control device <NUM> of <FIG> and <FIG>. For purposes of illustration, the techniques of <FIG> are described within the context of control device <NUM> of <FIG> and <FIG>, although computing devices having configurations different than that of control device <NUM> may perform the techniques of <FIG>. As one example, one or more processors of display device <NUM> of <FIG> may perform the techniques of <FIG>. As another example, one or more processors of wearable <NUM> of <FIG> and <FIG> may perform the techniques of <FIG>. As another example, one or more processors of a mobile computing device other than display device <NUM> and wearable <NUM> may perform the techniques of <FIG>.

Control device <NUM> may output, for display by a display device, a graphical user interface (<NUM>). For instance, processors <NUM> of control device <NUM> may execute an application module of application modules <NUM> and the executed application module may, with UI module <NUM>, generate instructions to render the graphical user interface. Processors <NUM> may execute communication module <NUM> to cause communication units <NUM> to output the instructions to generate the graphical user interface to display device <NUM>.

Control device <NUM> may determine, based on motion data, an estimated location of a wearable computing device (<NUM>). For instance, control device <NUM> may wirelessly receive acceleration data generated by accelerometers of motion sensors <NUM> of wearable <NUM>, and LEM <NUM> of control device <NUM> may perform a double integration of acceleration data, and add the resulting value to a previously known location of wearable <NUM>.

Control device <NUM> may output, for display by the display device and at the estimated location, a graphical pattern including a center region and a plurality of border regions (<NUM>). For instance, LRM <NUM> may insert graphical pattern <NUM> into the graphical user interface generated by the application module (e.g., such that the graphical pattern is overlaid on top of other elements of the graphical user interface).

Control device <NUM> may determine, based on optical data generated by an optical sensor of the wearable computing device, a refined location of the wearable computing device. For instance, LRM <NUM> may process optical data generated by an optical sensor of optical sensors <NUM> of wearable <NUM> to determine if the optical data indicates matches an optical characteristic of a region of graphical pattern <NUM>. Responsive to determining that a center region is detected in the optical data ("Yes" branch of <NUM>), LRM <NUM> may reduce a display size of graphical pattern <NUM> (<NUM>). Responsive to determining that neither the center region or any border regions are detected in the optical data ("No" branch of <NUM>), LRM <NUM> may increase a display size of graphical pattern <NUM> (<NUM>). Responsive to determining that a border region is detected in the optical data ("Yes" branch of <NUM>), LRM <NUM> may adjust the estimated location based on an adjustment direction associated with the detected border region (<NUM>).

Control device <NUM> may determine whether the wearable device has moved (<NUM>). For instance, control device <NUM> may determine, based on motion data generated by motion sensors <NUM>, whether an acceleration of wearable computing device <NUM> is greater than an acceleration threshold. Responsive to determining that the wearable device has moved ("Yes" branch of <NUM>), control device <NUM> may determine, based on motion data that represents the movement, an updated estimated location of a wearable computing device (<NUM>). Responsive to determining that the wearable device has not moved ("No" branch of <NUM>), LRM <NUM> may continue to perform the refinement operations (<NUM>-<NUM>).

As discussed above, the graphical pattern may be displayed overlaid on other elements of the graphical user interface. In some examples, control device <NUM> may perform one or more operations to reduce the likelihood that the graphical pattern will be "confused" for other elements of the graphical user interface. For instance, while causing the display device to display the graphical pattern, control device <NUM> may causing the display device to blink the graphical pattern at a predetermined rate (e.g., every <NUM>, <NUM>, <NUM>, <NUM>, <NUM> frames).

<FIG> are example graphical patterns that may be used for global initialization, in accordance with one or more techniques of this disclosure. As discussed above, in some examples, control device <NUM> may determine an initial location of wearable <NUM> by displaying a graphical pattern (e.g., graphical pattern <NUM> of <FIG>) at a particular location on display device <NUM> (e.g., the particular location functions as the initial location). In other examples, control device <NUM> may be configured to determine an initial location of wearable <NUM> on display device <NUM> by displaying various graphical patterns across an entirety of display device <NUM> (whereas graphical pattern <NUM> takes up less than all of display device <NUM>). Some example graphical patterns are shown in <FIG>. To perform the global initialization technique using the graphical patterns shown in <FIG>, control device <NUM> flash one or more of graphical patterns <NUM>, <NUM>, and <NUM> on the whole display area of display device <NUM> (e.g., such that a portion of the graphical pattern will be at the location of wearable <NUM>, regardless of where on display device <NUM> the user has placed wearable <NUM>).

The following numbered examples may illustrate one or more aspects of this disclosure:.

The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. A control unit including hardware may also perform one or more of the techniques of this disclosure.

Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various techniques described in this disclosure. Depiction of different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be realized by separate hardware, firmware, or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware, firmware, or software components, or integrated within common or separate hardware, firmware, or software components.

The techniques described in this disclosure may also be embodied or encoded in an article of manufacture including a computer-readable storage medium encoded with instructions. Instructions embedded or encoded in an article of manufacture including a computer-readable storage medium encoded, may cause one or more programmable processors, or other processors, to implement one or more of the techniques described herein, such as when instructions included or encoded in the computer-readable storage medium are executed by the one or more processors. Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a compact disc ROM (CD-ROM), a floppy disk, a cassette, magnetic media, optical media, or other computer readable media. In some examples, an article of manufacture may include one or more computer-readable storage media.

In some examples, a computer-readable storage medium may include a non-transitory medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM or cache).

Claim 1:
A method comprising:
causing a display device that is not presence-sensitive to output, at an initial location on the display device, a graphical pattern comprising a plurality of regions that each have a different optical characteristic, wherein the plurality of regions includes a center region and a plurality of border regions dispersed about the center region, wherein each border region is associated with a corresponding adjustment direction;
determining, by one or more processors and based on optical data generated by an optical sensor included in a photoplethysmogram sensor of a wearable computing device configured to be worn on a wrist of a user, a refined location of the wearable computing device, wherein determining the refined location comprises:
determining, based on the optical data, whether any of the plurality of regions are detected;
responsive to determining that a particular border region of the plurality of border regions is detected, adjusting, in the adjustment direction corresponding to the particular border region, the initial location to obtain the refined location; and
utilizing, by the one or more processors, the refined location of the wearable computing device as a location of user input for an application outputting a graphical user interface via the display device.