Patent Description:
Electronic devices such as computers can be controlled using computer mice and other input accessories. In virtual reality systems, force-feedback gloves can be used to control virtual objects. Cellular telephones may have touch screen displays and vibrators that are used to create haptic feedback in response to touch input.

Devices such as these may not be convenient for a user, may be cumbersome or uncomfortable, or may provide inadequate feedback. <CIT> discloses a glove interface object comprising magnetic sensors to determine the position of the glove in an interactive environment. <CIT> discloses a wearable remote interface device in the shape of a ring. <CIT> discloses a sensor circuit for finger mounted electronic devices. Mounted on the top and the sides of a finger leaving the fingertip free.

Dependent claims define the preferred embodiments.

Electronic devices that are configured to be mounted on the body of a user may be used to gather user input and to provide a user with output. For example, electronic devices that are configured to be worn on one or more of a user's fingers, which are sometimes referred to as finger devices or finger-mounted devices, may be used to gather user input and to supply output. A finger device may, as an example, include an inertial measurement unit with an accelerometer for gathering information on figure motions such as finger taps or free-space finger gestures, may include force sensors for gathering information on normal and shear forces in the finger device and the user's finger, and may include other sensors for gathering information on the interactions between the finger device (and the user's finger on which the device is mounted) and the surrounding environment. The finger device may include a haptic output device to provide the user's finger with haptic output and may include other output components.

One or more finger devices may gather user input from a user. The user may use finger devices in operating a virtual reality or mixed reality device (e.g., head-mounted equipment such as glasses, goggles, a helmet, or other device with a display). During operation, the finger devices may gather user input such as information on interactions between the finger device(s) and the surrounding environment (e.g., interactions between a user's fingers and the environment, including finger motions and other interactions associated with virtual content displayed for a user). The user input may be used in controlling visual output on the display. Corresponding haptic output may be provided to the user's fingers using the finger devices. Haptic output may be used, for example, to provide the fingers of a user with a desired texture sensation as a user is touching a real object or as a user is touching a virtual object. Haptic output can also be used to create detents and other haptic effects.

Finger devices can be worn on any or all of a user's fingers (e.g., the index finger, the index finger and thumb, three of a user's fingers on one of the user's hands, some or all fingers on both hands, etc.). To enhance the sensitivity of a user's touch as the user interacts with surrounding objects, finger devices may have inverted U shapes or other configurations that allow the finger devices to be worn over the top and sides of a user's finger tips while leaving the user's finger pads exposed. This allows a user to touch objects with the finger pad portions of the user's fingers during use. If desired, finger devices may be worn over knuckles on a user's finger, between knuckles, and/or on other portions of a user's finger. The use of finger devices on a user's finger tips is sometimes described herein as an example.

Users can use the finger devices to interact with any suitable electronic equipment. For example, a user may use one or more finger devices to interact with a virtual reality or mixed reality system (e.g., a head-mounted device with a display), to supply input to a desktop computer, tablet computer, cellular telephone, watch, ear buds, or other accessory, or to interact with other electronic equipment.

<FIG> is a schematic diagram of an illustrative system of the type that may include one or more finger devices. As shown in <FIG>, system <NUM> may include electronic device(s) such as finger device(s) <NUM> and other electronic device(s) <NUM>. Each finger device <NUM> may be worn on a finger of a user's hand. Additional electronic devices in system <NUM> such as devices <NUM> may include devices such as a laptop computer, a computer monitor containing an embedded computer, a tablet computer, a desktop computer (e.g., a display on a stand with an integrated computer processor and other computer circuitry), a cellular telephone, a media player, or other handheld or portable electronic device, a smaller device such as a wristwatch device, a pendant device, a headphone or earpiece device, a head-mounted device such as glasses, goggles, a helmet, or other equipment worn on a user's head, or other wearable or miniature device, a television, a computer display that does not contain an embedded computer, a gaming device, a remote control, a navigation device, an embedded system such as a system in which equipment is mounted in a kiosk, in an automobile, airplane, or other vehicle, a removable external case for electronic equipment, a strap, a wrist band or head band, a removable cover for a device, a case or bag that has straps or that has other structures to receive and carry electronic equipment and other items, a necklace or arm band, a wallet, sleeve, pocket, or other structure into which electronic equipment or other items may be inserted, part of a chair, sofa, or other seating (e.g., cushions or other seating structures), part of an item of clothing or other wearable item (e.g., a hat, belt, wrist band, headband, sock, glove, shirt, pants, etc.), or equipment that implements the functionality of two or more of these devices.

With one illustrative configuration, which may sometimes be described herein as an example, device <NUM> is a finger-mounted device having a finger-mounted housing with a U-shaped body that grasps a user's finger or a finger-mounted housing with other shapes configured to rest against a user's finger and device(s) <NUM> is a cellular telephone, tablet computer, laptop computer, wristwatch device, head-mounted device, a device with a speaker, or other electronic device (e.g., a device with a display, audio components, and/or other output components). A finger device with a U-shaped housing may have opposing left and right sides that are configured to receive a user's finger and a top housing portion that couples the left and right sides and that overlaps the user's fingernail.

Devices <NUM> and <NUM> may include control circuitry <NUM> and <NUM>. Control circuitry <NUM> and <NUM> may include storage and processing circuitry for supporting the operation of system <NUM>. The storage and processing circuitry may include storage such as nonvolatile memory (e.g., flash memory or other electrically-programmable-read-only memory configured to form a solid state drive), volatile memory (e.g., static or dynamic random-access-memory), etc. Processing circuitry in control circuitry <NUM> and <NUM> may be used to gather input from sensors and other input devices and may be used to control output devices. The processing circuitry may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors and other wireless communications circuits, power management units, audio chips, application specific integrated circuits, etc..

To support communications between devices <NUM> and <NUM> and/or to support communications between equipment in system <NUM> and external electronic equipment, control circuitry <NUM> may communicate using communications circuitry <NUM> and/or control circuitry <NUM> may communicate using communications circuitry <NUM>. Circuitry <NUM> and/or <NUM> may include antennas, radio-frequency transceiver circuitry, and other wireless communications circuitry and/or wired communications circuitry. Circuitry <NUM> and/or <NUM>, which may sometimes be referred to as control circuitry and/or control and communications circuitry, may, for example, support bidirectional wireless communications between devices <NUM> and <NUM> over wireless link <NUM> (e.g., a wireless local area network link, a near-field communications link, or other suitable wired or wireless communications link (e.g., a Bluetooth® link, a WiFi® link, a <NUM> link or other millimeter wave link, etc.). Devices <NUM> and <NUM> may also include power circuits for transmitting and/or receiving wired and/or wireless power and may include batteries. In configurations in which wireless power transfer is supported between devices <NUM> and <NUM>, in-band wireless communications may be supported using inductive power transfer coils (as an example).

Devices <NUM> and <NUM> may include input-output devices such as devices <NUM> and <NUM>. Input-output devices <NUM> and/or <NUM> may be used in gathering user input, in gathering information on the environment surrounding the user, and/or in providing a user with output. Devices <NUM> may include sensors <NUM> and devices <NUM> may include sensors <NUM>. Sensors <NUM> and/or <NUM> may include force sensors (e.g., strain gauges, capacitive force sensors, resistive force sensors, etc.), audio sensors such as microphones, touch and/or proximity sensors such as capacitive sensors, optical sensors such as optical sensors that emit and detect light, ultrasonic sensors, and/or other touch sensors and/or proximity sensors, monochromatic and color ambient light sensors, image sensors, sensors for detecting position, orientation, and/or motion (e.g., accelerometers, magnetic sensors such as compass sensors, gyroscopes, and/or inertial measurement units that contain some or all of these sensors), muscle activity sensors (EMG) for detecting finger actions, radio-frequency sensors, depth sensors (e.g., structured light sensors and/or depth sensors based on stereo imaging devices), optical sensors such as self-mixing sensors and light detection and ranging (lidar) sensors that gather time-of-flight measurements, optical sensors such as visual odometry sensors that gather position and/or orientation information using images gathered with digital image sensors in cameras, gaze tracking sensors, visible light and/or infrared cameras having digital image sensors, humidity sensors, moisture sensors, and/or other sensors. In some arrangements, devices <NUM> and/or <NUM> may use sensors <NUM> and/or <NUM> and/or other input-output devices <NUM> and/or <NUM> to gather user input (e.g., buttons may be used to gather button press input, touch sensors overlapping displays can be used for gathering user touch screen input, touch pads may be used in gathering touch input, microphones may be used for gathering audio input, accelerometers may be used in monitoring when a finger contacts an input surface and may therefore be used to gather finger press input, etc.). If desired, device <NUM> and/or device <NUM> may include rotating buttons (e.g., a crown mechanism on a watch or finger device or other suitable rotary button that rotates and that optionally can be depressed to select items of interest). Alphanumeric keys and/or other buttons may be included in devices <NUM> and/or <NUM>.

Devices <NUM> and/or <NUM> may include haptic output devices <NUM> and/or <NUM>. Haptic output devices <NUM> and/or <NUM> can produce motion that is sensed by the user (e.g., through the user's fingertips). Haptic output devices <NUM> and/or <NUM> may include actuators such as electromagnetic actuators, motors, piezoelectric actuators, electroactive polymer actuators, vibrators, linear actuators, rotational actuators, actuators that bend bendable members, actuator devices that create and/or control repulsive and/or attractive forces between devices <NUM> and/or <NUM> (e.g., components for creating electrostatic repulsion and/or attraction such as electrodes, components for producing ultrasonic output such as ultrasonic transducers, components for producing magnetic interactions such as electromagnets for producing directcurrent and/or alternating-current magnetic fields, permanent magnets, magnetic materials such as iron or ferrite, and/or other circuitry for producing repulsive and/or attractive forces between devices <NUM> and/or <NUM>). In some situations, actuators for creating forces in device <NUM> may be used in squeezing a user's finger and/or otherwise directly interacting with a user's finger pulp. In other situations, these components may be used to interact with each other (e.g., by creating a dynamically adjustable electromagnetic repulsion and/or attraction force between a pair of devices <NUM> and/or between device(s) <NUM> and device(s) <NUM> using electromagnets).

If desired, input-output devices <NUM> and/or <NUM> may include other devices <NUM> and/or <NUM> such as displays (e.g., in device <NUM> to display images for a user), status indicator lights (e.g., a light-emitting diode in device <NUM> and/or <NUM> that serves as a power indicator, and other light-based output devices), speakers and other audio output devices, electromagnets, permanent magnets, structures formed from magnetic material (e.g., iron bars or other ferromagnetic members that are attracted to magnets such as electromagnets and/or permanent magnets), batteries, etc. Devices <NUM> and/or <NUM> may also include power transmitting and/or receiving circuits configured to transmit and/or receive wired and/or wireless power signals.

<FIG> is a top view of a user's finger (finger <NUM>) and an illustrative finger-mounted device <NUM>. As shown in <FIG>, device <NUM> may be formed from a finger-mounted unit that is mounted on or near the tip of finger <NUM> (e.g., partly or completely overlapping fingernail <NUM>). If desired, device <NUM> may be worn elsewhere on a user's fingers such as over a knuckle, between knuckles, etc. Configurations in which a device such as device <NUM> is worn between fingers <NUM> may also be used.

A user may wear one or more of devices <NUM> simultaneously. For example, a user may wear a single one of devices <NUM> on the user's ring finger or index finger. As another example, a user may wear a first device <NUM> on the user's thumb, a second device <NUM> on the user's index finger, and an optional third device <NUM> on the user's middle finger. Arrangements in which devices <NUM> are worn on other fingers and/or all fingers of one or both hands of a user may also be used.

Control circuitry <NUM> (and, if desired, communications circuitry <NUM> and/or input-output devices <NUM>) may be contained entirely within device <NUM> (e.g., in a housing for a fingertip-mounted unit) and/or may include circuitry that is coupled to a fingertip structure (e.g., by wires from an associated wrist band, glove, fingerless glove, etc.). Configurations in which devices <NUM> have bodies that are mounted on individual user fingertips are sometimes described herein as an example.

<FIG> is a cross-sectional side view of an illustrative finger device (finger-mounted device) <NUM> showing illustrative mounting locations <NUM> for electrical components (e.g., control circuitry <NUM>, communications circuitry <NUM>, and/or input-output devices <NUM>) within and/or on the surface(s) of finger device housing <NUM>. These components may, if desired, be incorporated into other portions of housing <NUM>.

As shown in <FIG>, housing <NUM> may have a U shape (e.g., housing <NUM> may be a U-shaped housing structure that faces downwardly and covers the upper surface of the tip of user finger <NUM> and fingernail <NUM>). During operation, a user may press against structures such as structure <NUM>. As the bottom of finger <NUM> (e.g., finger pulp 40P) presses against surface <NUM> of structure <NUM>, the user's finger may compress and force portions of the finger outwardly against the sidewall portions of housing <NUM> (e.g., for sensing by force sensors or other sensors mounted to the side portions of housing <NUM>). Lateral movement of finger <NUM> in the X-Y plane may also be sensed using force sensors or other sensors on the sidewalls of housing <NUM> or other portions of housing <NUM> (e.g., because lateral movement will tend to press portions of finger <NUM> against some sensors more than others and/or will create shear forces that are measured by force sensors that are configured to sense shear forces).

Ultrasonic sensors, optical sensors, inertial measurement units, strain gauges and other force sensors, radio-frequency sensors, and/or other sensors may be used in gathering sensor measurements indicative of the activities of finger <NUM>. If desired, these sensors may also be used in mapping the contours of three-dimensional objects (e.g., by time-of-flight measurements and/or other measurements). For example, an ultrasonic sensor such as a two-dimensional image sensor or an ultrasonic sensor with a single ultrasonic transducer element may emit free-space ultrasonic sound signals that are received and processed after reflecting off of external objects. This allows a three-dimensional ultrasonic map to be generated indicating the shapes and locations of the external objects.

In some configurations, finger activity information (position, movement, orientation, etc.) may be gathered using sensors that are mounted in external electronic equipment (e.g., in a computer or other desktop device, in a head-mounted device or other wearable device, and/or in other electronic device <NUM> that is separate from device <NUM>). For example, optical sensors such as images sensors that are separate from devices <NUM> may be used in monitoring devices <NUM> to determine their position, movement, and/or orientation. If desired, devices <NUM> may include passive and/or active optical registration features to assist an image sensor in device <NUM> in tracking the position, orientation, and/or motion of device <NUM>. For example, devices <NUM> may include light-emitting devices such as light-emitting diodes and/or lasers. The light-emitting devices may be arranged in an asymmetric pattern on housing <NUM> and may emit light that is detected by an image sensor, depth sensor, and/or other light-based tracking sensor circuitry in device <NUM>. By processing the received patterned of emitted light, device <NUM> can determine the position, orientation, and/or motion of device <NUM>.

Tracking can also be performed that involves extrapolating from a known body part orientation (e.g., a finger orientation) to produce orientation information on other body parts (e.g., wrist and/or arm orientation estimated using inverse kinematics). Visual odometry sensors may, if desired, be included in devices <NUM>. These sensors may include image sensors that gather frames of image data of the surroundings of devices <NUM> and may be used in measuring position, orientation, and/or motion from the frame of image data. Lidar, ultrasonic sensors oriented in multiple directions, radio-frequency tracking sensors, and/or other finger device tracking arrangements may be used, if desired. In some arrangements, user input for controlling system <NUM> can include both user finger input and other user input (e.g., user eye gaze input, user voice input, etc.). For example, gaze tracking information such as a user's point-of-gaze measured with a gaze tracker can be fused with finger input when controlling device <NUM> and/or devices <NUM> in system <NUM>. The finger input may include information on finger orientation, position, and/or motion and may include information on how forcefully a finger is pressing against surfaces (e.g., force information).

The sensors in device <NUM> may, for example, measure how forcefully a user is moving device <NUM> (and finger <NUM>) against surface <NUM> (e.g., in a direction parallel to the surface normal n of surface <NUM> such as the -Z direction of <FIG>) and/or how forcefully a user is moving device <NUM> (and finger <NUM>) within the X-Y plane, tangential to surface <NUM>. The direction of movement of device <NUM> in the X-Y plane and/or in the Z direction can also be measured by the force sensors and/or other sensors <NUM> at locations <NUM>.

Structure <NUM> may be a portion of a housing of device <NUM>, may be a portion of another device <NUM> (e.g., another housing <NUM>), may be a portion of a user's finger <NUM> or other body part, may be a surface of a real-world object such as a table, a movable real-world object such as a bottle or pen, or other inanimate object external to device <NUM>, and/or may be any other structure that the user can contact with finger <NUM> while moving finger <NUM> in a desired direction with a desired force. Because motions such as these can be sensed by device <NUM>, device(s) <NUM> can be used to gather pointing input (e.g., input moving a cursor or other virtual object on a display such as a display in devices <NUM>), can be used to gather tap input, swipe input, pinch-to-zoom input (e.g., when a pair of devices <NUM> is used), or other gesture input (e.g., finger gestures, hand gestures, arm motions, etc.), and/or can be used to gather other user input.

<FIG> is a side view of an illustrative finger device on a finger of a user. In the illustrative configuration of <FIG>, device <NUM> includes touch sensor <NUM>. Touch sensor <NUM> may be formed from an array of capacitive touch sensor electrodes such as electrodes <NUM> overlapping the side and/or top surfaces of housing <NUM>. Touch sensor <NUM> may be used to gather touch input such as input from direct contact and/or close proximity with a different finger of the user or other external object. In the example of <FIG>, touch sensor <NUM> may overlap touch input area <NUM> on the side(s) of device <NUM>. If desired, additional touch input may be gathered in adjacent areas such as touch input area <NUM> on the exposed side of finger <NUM> adjacent to device <NUM>. Touch input may be gathered from area <NUM> using sensors in device <NUM> that are directed towards area <NUM>. These sensors may be, for example, capacitive sensors, optical sensors, ultrasonic sensors, and/or other sensors that can monitor area <NUM> (see, e.g., sensors <NUM>). With one illustrative configuration, sensors <NUM> are optical sensors having light emitters (e.g., light-emitting diodes or lasers) that emit light <NUM> overlapping area <NUM> and having light detectors (e.g., photodiodes) that measure reflected light from a user's finger or other external object in area <NUM>. In this way, the area covered by the touch sensor circuitry of device <NUM> can extend across both portions of housing <NUM> and portions of adjacent body parts such as area <NUM> on finger <NUM>.

System <NUM> may have head-mounted display devices or other devices that present mixed reality content to a user. In a mixed reality environment, virtual content (e.g., computer-generated content from control circuitry in system <NUM>) may be overlaid on real-world content (e.g., real-world images obtained from directly viewing the real world through an optical coupler and/or real-world images obtained with a front-facing camera or other image sensor that is operated in a pass-through mode to provide real-world images on a display for the user. The mixed reality content of system <NUM> may, as an example, include icons and other computer-generated visual content (sometimes referred to as virtual content) that is overlaid over areas that gather user input (e.g., touch input). Consider, as an example, the scenario of <FIG>. In this example, a user is wearing finger device <NUM> on finger <NUM>. Touch sensor <NUM> is formed in device <NUM> (e.g., in housing <NUM>) and gathers touch sensor input in area <NUM> on the outer surface of housing <NUM>. Optical sensors <NUM> gather touch input from area <NUM> on finger <NUM> adjacent to device <NUM>. The user may be viewing finger <NUM> and device <NUM> through a mixed reality head-mounted device (see, e.g., device <NUM> of <FIG>). The mixed reality device may present computer-generated content (virtual content generated by device <NUM>) that overlaps real-world content in virtual content area <NUM>.

The virtual content in area <NUM> may include, for example, selectable icons corresponding to functions (e.g., functions performed by application software and/or operating system software) that a user may invoke by supplying corresponding touch input. For example, device <NUM> may be a mixed reality head-mounted device that presents a selectable calendar icon in area <NUM>. When the user uses a free finger other than the finger on which device <NUM> is being worn to touch the calendar icon in area <NUM>, this touch input will be detected by touch sensor <NUM> or optical sensors <NUM>, depending on the portion of area <NUM> in which the calendar item was presented. In response to user touch input selection of the calendar icon, control circuitry in device <NUM> can take suitable action such as launching a calendar application and presenting content for the launched calendar application visually using the display in device <NUM> (e.g., as an overlay over an image of the real world). If desired, free-form touch input may be gathered (e.g., a user may trace the shape of an alphanumeric character in area <NUM> that serves as an input command). Haptic output may be provided by device <NUM> in response to the received touch input.

As another example, a volume slider or other sliding control icon may be displayed in region <NUM>. As the user's finger points at and overlaps the sliding control icon, the sliding control icon is selected. The user can then slide the user's finger back and forth in region <NUM> to adjust the slider. If desired, haptic output (e.g., a click) can be provided in response to selection and/or movement of the control icon. For example, haptic output detents (vibrations that are supplied when the user's finger position coincides with predetermined locations <NUM>) may be supplied during user finger interactions in area <NUM>. In some arrangements, an icon (e.g., a dot, a glowing dot, a blinking dot, a finger icon, a movable line, etc.) may be used to depict a finger location in region <NUM>. In general, any suitable content may be displayed in areas such as area <NUM> (e.g., areas that overlap all or part of device <NUM>, areas that overlap all or part of finger <NUM>, and/or other areas that overlap real-world objects). This content may include still and/or moving images containing text, graphics, photographs, real-world video, moving animations, icons, and/or other content).

<FIG> is a front view of device <NUM> showing how deice <NUM> may have a touch sensor (illustrative touch sensor <NUM>) that overlaps both sidewalls (sides) 44SW and top portion 44T of housing <NUM>. Touch sensor <NUM> may have electrodes formed on a flexible printed circuit, on an inner surface of a housing sidewall in housing <NUM>, on an exterior surface of housing <NUM>, and/or other suitable portions of device <NUM>. In one illustrative configuration, circuitry such as circuitry <NUM> in housing <NUM> (e.g., a printed circuit, integrated circuits, touch sensor control circuitry, etc.) may communicate with touch sensors <NUM> on sidewalls 44SW of housing <NUM> (e.g., using signal paths formed from a flexible printed circuit that extends between circuitry <NUM> and sensors <NUM>, using a flexible printed circuit substrate that contains both signal paths for coupling to circuitry <NUM> and an array of capacitive touch sensor electrodes, etc.).

Flexible printed circuits containing capacitive touch sensor electrodes may be formed from a conductive material such as silver (e.g., silver paint/ink including silver filler in a polymer binder), silver nanowires, copper, or other metal traces on a polymer substrate material such as polyethylene terephthalate (PET) substrate or polyimide substrate. Configurations may also be used in which capacitor sensor substrates include fabric (e.g., fabric onto which conductive traces are deposited and/or fabric with conductive strands of material forming electrodes, etc.). Capacitive sensors may use mutual capacitance sensing arrangements or self-capacitance sensing arrangements. Touch sensors for device <NUM> may be formed on a single sidewall 44SW, one two sidewalls 44SW (e.g., housing portions on the left and right of finger <NUM>), on one or both sidewalls 44SW and top portion 44T, on top portion 44T only, and/or on any other suitable areas of device <NUM>.

If desired, non-capacitive touch sensing such as touch/proximity sensors based on optical sensing, ultrasonic sensing, force detection, etc. may be used in addition to and/or instead of using capacitive touch sensing. Touch sensors such as illustrative touch sensors <NUM> and/or <NUM> of <FIG> may be formed using single touch sensor elements (e.g., a single capacitive touch sensor electrode, a single ultrasonic sensor element, a single optical sensor element, etc.) and/or may be formed using one-dimensional (line-shaped) and/or two-dimensional arrays of sensor elements (e.g., sensors with rows and columns of capacitive sensing electrodes, light sensor devices, ultrasonic sensor devices, etc.).

A cross-sectional top view of device <NUM> in an illustrative configuration in which device <NUM> includes force sensor circuitry such as strain gauge circuitry and haptic output components is shown in <FIG>. As shown in <FIG>, electrodes <NUM> may form touch sensor <NUM>. For example, touch sensor <NUM> may be a two-dimensional touch sensor that extends over the sides and/or top of housing <NUM> (e.g., on the inner surface of housing <NUM> and/or other suitable portions of device <NUM>). Housing <NUM> may have an elongated portion such as portion <NUM>' of <FIG> (sometimes referred to as an arm) that extends along longitudinal housing axis <NUM> (e.g., an axis parallel to the length of finger <NUM>). Arm <NUM>' may be configured to bend in direction <NUM> under pressure from finger <NUM> (e.g., sideways finger motion and/or other finger input). Strain gauge circuitry <NUM> may be formed near the base of arm <NUM>' (e.g., where arm <NUM>' attaches to the remainder of housing <NUM>) so that bending of arm <NUM>' can be detected by strain gauge circuitry <NUM>. Strain gauge circuitry <NUM> may contain one or more strain gauges. If desired, other force sensing components may be used to detect bending of arm <NUM>' (e.g., a piezoelectric force sensor, etc.).

As shown in <FIG>, additional components such as component <NUM> may be mounted on arm <NUM>'. Component <NUM> may be, as an example, a haptic output device that generates haptic output for finger <NUM> or other suitable electrical component. In some arrangements, component <NUM> may be a piezoelectric device that serves both as a haptic output component and as a force sensor. Using circuitry of the type shown in <FIG>, device <NUM> can monitor finger forces imposed on the sidewalls of device <NUM>, thereby measuring shear and/or normal forces as a user manipulates device <NUM> and interacts with the surrounding environment. Touch input can be gathered from touch sensor <NUM> and/or can be extrapolated from force sensor measurements made with strain gauge circuitry <NUM>.

<FIG> shows how housing <NUM> may be provided with one or more sensors <NUM> that are used to sense the separation between device <NUM> (e.g., housing <NUM> and finger <NUM>) and surface <NUM> of structure <NUM>. Sensors <NUM> may be optical sensors, capacitive sensors, ultrasonic sensors, and/or other sensors (e.g., proximity sensors, distance sensors, etc.). Using sensors such as sensor <NUM> of <FIG>, device <NUM> can monitor the distance between device <NUM> and surface <NUM>, and/or can gather information on device orientation, and/or device motion.

<FIG> shows how housing <NUM> may be provided with a depth sensor or other sensor having multiple sensor components <NUM>. Sensor components <NUM> may be, for example, a pair of digital image sensors configured to form a stereoscopic depth sensor. With one illustrative configuration, portion <NUM> of housing <NUM> may be coupled to a hinge that is aligned with hinge axis <NUM>. Portion <NUM> may be rotated (pivoted) about hinge axis <NUM> in direction <NUM> when it is desired to use components <NUM> (e.g., to capture images of objects in directions <NUM>). Using the depth sensor of <FIG>, device <NUM> can gather information on the orientation, position, and/or motion of device <NUM> relative to other structures and/or can collect information about the surroundings of device <NUM> (e.g., images of real-world objects, three-dimensional maps of real-world objects, etc.).

<FIG> is a perspective view of device <NUM> in an illustrative configuration in which device <NUM> has a set of sensors <NUM> oriented to gather data in orthogonal directions. There may be, for example, three sensors <NUM>, each of which has its direction of operation (e.g., its most sensitive operating direction) oriented respectively along the X, Y, or Z axis of <FIG>. As device <NUM> is moved while being worn by a user on finger <NUM>, sensors <NUM> may be used to gather information on the movement, orientation, and position of device <NUM>. Sensors <NUM> may be, for example, radio-frequency sensors such as radio-frequency transceivers (e.g., transmitters and receivers) coupled to directional antennas. During operation, each sensor <NUM> may emit radio-frequency signals and may detect corresponding reflected signals from external objects. In another illustrative arrangement, base stations and/or other external electronic equipment (e.g., devices <NUM>, etc.) can emit reference (beacon) radio-frequency signals that are measured using the receivers in sensors <NUM>. If desired, other types of directional sensors may be include in device <NUM> (e.g., optical sensors such as lidar sensors, directional ultrasonic sensors, etc.). The use of three orthogonally oriented radio-frequency sensors is illustrative.

<FIG> is a top view of device <NUM> in an illustrative configuration in which housing <NUM> has a forwardly protruding portion such as portion <NUM> that extends past the tip of finger <NUM> when device <NUM> is being worn by a user. Portion <NUM> may serve as a support for electrical components such as component <NUM>. Component <NUM> may be, for example, a proximity sensor and/or distance sensor such as an optical sensor, a capacitive sensor, an ultrasonic sensor, and/or other sensor configured to gather information on the distance between device <NUM> and surrounding surfaces. This allows device <NUM> to gather information on the position, orientation, and/or movement of device <NUM> and/or information about nearby objects.

As shown in <FIG>, a user may have more than a single device <NUM>. In the <FIG> example, the user is wearing a first finger device <NUM> on a first finger <NUM> and a second finger device <NUM> on a second finger <NUM>. Each of these finger devices <NUM> may gather user input (e.g., input measured through force sensors, ultrasonic sensors, optical sensors, capacitive sensors, etc.) as the first and second fingers <NUM> interact with a third finger (e.g., the underside surface of the user's thumb in this example). The input that is gathered in this way may include information on interactions between finger devices <NUM> and surfaces associated with other fingers <NUM> and/or other surfaces (e.g., surfaces associated with a user's legs, the back of a user's hand, the palm of a user's hand, or other body parts, surfaces associated with inanimate external objects such as pencils, bottles, tabletops, etc., surfaces associated with electronic devices such as display surfaces, keyboard surfaces, housing and button surfaces in accessories, etc.). The input may include touch input (e.g., input indicative of contact between finger <NUM> and an external surface), shear force input (e.g., information indicating that a user is pushing and/or dragging finger <NUM> to one side while contacting an external surface with finger pad 40P), and/or normal force input (e.g., information on how forcefully finger <NUM> is pressing against an external surface).

In some arrangements, information on the relative motions between devices <NUM> may be gathered. For example, sensors in devices <NUM> may be used to gather information indicating that devices <NUM> are moving towards or away from each other and/or information on device position, orientation, etc. This allows users with multiple devices <NUM> to make multifinger gestures (e.g., pinch-to-zoom gestures, gestures in which an item is selected by grasping the item with opposing fingers, etc.).

In configurations of the type shown in <FIG> in which the user's fingers <NUM> with devices <NUM> are contacting and interacting with another finger <NUM> or other surface of the user's body, the user need not be in the vicinity of an external inanimate object such as a tabletop in order to provide input. Rather, the user may supply touch gestures and other gestures by creating finger interactions between the fingers <NUM> that are wearing devices <NUM> and/or fingers not wearing devices <NUM> and/or other objects. If desired, finger motions through the air and/or other finger activity associated with changes in finger location, orientation, motion, and/or finger forces may be gathered as finger input using devices <NUM>. Touch sensor input may also be gathered using touch sensor circuitry in devices <NUM>. The finger input may include finger taps, finger swipes (e.g., velocity-dependent and/or direction-dependent swipe gestures), finger pinch-to-zoom gestures, gestures in which fingers squeeze together, gestures in which fingers press with different forces against a surface, three-dimensional free space gestures such as finger flicks and/or up-and-down finger motions (e.g., a up motion followed by a down motion on a particular tap location within a predetermined time to select an item associated with the tap location), gestures such as thumb rolls, fine-tuning gestures, inertial swipes, tap-and-swipe gestures, dwell-time-dependent force input, and/or other finger activities in which finger(s) <NUM> move through the air, move along a surface, and/or press against a surface in a predetermined pattern.

System <NUM> may include an optical sensor such as a gaze detection sensor (sometimes referred to as a gaze detector, gaze tracker, gaze tracking system, or eye monitoring system). A gaze tracking system for system <NUM> may, for example, include image sensors, light sources, and/or other equipment that is used in monitoring the eyes of a user. This system may include one or more visible and/or infrared cameras that face a viewer's eyes and capture images of the viewer's (user's) eyes. During operation of system <NUM>, control circuitry in system <NUM> (e.g., control circuitry coupled to a housing in device <NUM>) may use the gaze tracking system to track a viewer's gaze. Cameras and/or other sensors in device <NUM> may, for example, determine the location of a user's eyes (e.g., the centers of the user's pupils) and may determine the direction in which the user's eyes are oriented.

The orientation of the user's gaze may be used to determine a location in a virtual and/or real-world environment in which a user's eyes are directed (sometimes referred to as the user's point-of-gaze). If desired, device <NUM> and/or other equipment in system <NUM> may use gaze tracking information such as information on the user's point-of-gaze in determining which actions to take in system <NUM>. For example, a gaze tracking system may determine that a user's point-of-gaze is directed towards a first object and not a second object and may respond by assuming that the user is visually selecting the first object and not the second object. Finger input and/or other user input may be used in combination with input such as point-of-gaze information in determining which actions are to be taken in system <NUM>.

Consider, as an example, a scenario of the type shown in <FIG>. In this example, device <NUM> has a housing in which gaze tracker <NUM> has been mounted for monitoring a user's eyes <NUM>. Device <NUM> may include components such as component <NUM>. Component <NUM> may be, for example, a display that is configured to display images for the user. The image may include one or more objects (e.g., visual items) such as object <NUM>. Control circuitry in device <NUM> may use gaze tracker <NUM> to determine the direction <NUM> in which the user is viewing component <NUM> or other object. Using direction <NUM> and/or other information from gaze tracker <NUM> and/or other sensors (e.g., a depth sensor and/or other sensors that determine the distance of the user from device <NUM>), device <NUM> may determine the location of the user's point-of-gaze <NUM> on component <NUM>. For example, device <NUM> can determine whether a virtual object such as object <NUM> on the display of <FIG> is currently being viewed by the user.

Another illustrative system with gaze tracking is shown in <FIG>. In the example of <FIG>, device <NUM> is a head-mounted device having a head-mounted support structure <NUM> (sometimes referred to as a housing) that is configured to be worn on the head of a user. Rear facing gaze tracking system <NUM> may monitor user's eyes <NUM> to determine the direction <NUM> of the user's gaze. Additional sensors (e.g. depth sensor <NUM>) may be used in determining the location and/or other attributes of objects in the user's field of view such as object <NUM> of <FIG>. Object <NUM> may be a real-world object (e.g., a body part of the user or other person, an inanimate object with circuitry such as one or more devices <NUM>, a non-electronic inanimate object such as a pencil, ball, bottle, cup, table, wall, etc.) or may be a computer-generated (virtual) object that is being presented to the user's eyes <NUM> by a display in device <NUM> (e.g., a see-through display system or a display system in which virtual content is overlaid on real-world images on the display that have been captured with camera <NUM>). Using information on the direction <NUM> of the user's gaze and information on the relative position between the user and object <NUM> (e.g., information from a depth sensor in device <NUM> and/or information on virtual objects being presented to the user), device <NUM> may determine when the user's point-of-gaze <NUM> coincides with object <NUM>.

Arrangements of the type shown in <FIG> allow a user to interact with real-world content and computer-generated (virtual) content. For example, a user may select an object of interest by directing point-of-gaze <NUM> towards that object (e.g., for more than a predetermined dwell time and/or until associated user input such as finger input is received to confirm selection). Using finger device(s) <NUM> and/or other equipment in system <NUM>, the user may perform operations on the selected object.

Consider, as a first example, a scenario in which object <NUM> is a computer-generated icon. In this situation, after aligning point-of-gaze <NUM> to overlap the computer-generated icon and thereby select the icon for further action, a user may supply a command with finger devices <NUM> and/or other input components in system <NUM> that direct system <NUM> to commence an associated operation in system <NUM>. If, as an example, the icon is an email icon, system <NUM> may, upon receipt of finger input from the user, launch an email program on device <NUM>.

In a second example, object <NUM> is a real-world object such as a non-electronic inanimate object (e.g., an object being viewed by the user of device <NUM> of <FIG> while device <NUM> is being worn on the head of the user). In response to detecting that the user's point-of-gaze <NUM> is directed at object <NUM> and in response to receipt of user finger input or other suitable input from the user, device <NUM> may generate a virtual object that overlaps all or part of object <NUM> in the user's field of view. Other operations may include, magnifying part of object <NUM>, changing the color or texture of object <NUM>, adding an outline around object <NUM>, adding graphical elements that are aligned with object <NUM>, and/or taking other suitable actions.

In a third example, object <NUM> is a real-world object that includes circuitry. Object <NUM> may be, for example, a wireless speaker or other electronic device <NUM>. In response to detecting that the user's point-of-gaze <NUM> is directed at object <NUM> and in response to receipt of user finger input or other suitable input from the user, device <NUM> may adjust the output volume of the speaker. If the object that coincides with point-of-gaze <NUM> is a device such as a television, the channel of the television may be changed in response to the user finger input or other input. In this way, a user can interact with electronic devices around the user's home or office or electronic devices in other environments simply by gazing at the objects and supplying additional user input in coordination with this point-of-gaze selection. The additional user input may be made using finger device(s) <NUM> and/or other input devices (e.g., a mouse or other pointing device, a device that processes voice commands, a keyboard, a touch screen display in a cellular telephone or tablet device, etc.). Point-of-gaze dwell time, eye blinks, and other eye activity may also be used as a user input.

A user may touch items in the user's surroundings while wearing finger devices <NUM>. Measurements made with sensors in devices <NUM> as the user touches the surfaces of these items can be used in determining the contours of the items. This information can then be combined with optional additional sensor data such as depth sensor data, camera images, etc. to determine the attributes of the items such as item size, item shape, item location, etc. Examples of sensors that may be used in devices <NUM> to measure the contours of items include inertial measurement units, which can track the orientation, position, and/or movement of devices <NUM> in three dimensions and force and/or touch sensors in devices <NUM> that can sense when a user has contacted the surface of an item. Depth sensors in devices <NUM> and/or <NUM> may also be used in gathering three-dimensional surface maps (surface contour information) for objects in the user's surroundings.

Once system <NUM> has gathered information on the contour of an object in the user's surroundings, virtual content may be created that partially or completely overlaps the surface of the object and/or haptic output associated with the surface of the object may be generated. As an example, knowing the locations of the surfaces of a cube, system <NUM> can overlay virtual content on one or more cube surfaces and can use haptic output devices in finger devices <NUM> to apply haptic output to the user's fingers <NUM> whenever system <NUM> determines that the user's fingers are touching these surfaces. If desired, different surfaces of the cube or other object can be provided with different virtual textures using the haptic output devices. As an example, control circuitry in device <NUM> can supply a first drive signal D with a relatively high amplitude and frequency whenever the user's fingers are touching a first side of the cube (see, e.g., drive signal D of time period <NUM> in the graph of <FIG> in which drive signal D has been plotted as a function of time t). Whenever the user's fingers are determined to be touching a second side of the cube, a different texture can be supplied to the user's fingers with the haptic output components in finger devices <NUM>. As an example, if the user touches the second side of the cube during time period <NUM> of <FIG>, a lower-frequency and lowermagnitude drive signal D can be used to control the haptic output components in finger devices <NUM>. As a result, the user will feel a first texture when touching the first side of the cube and a second texture when touching a second side of the cube. If desired, cubes and other objects can be provided with haptic effects along the edges of the objects such as haptic output can create shape and/or rounded edges, can create effects associated with compliant structures, and/or can generate detents, force-feedback simulating motion resistance, clicks simulating depression and/or release of a button with a physical click sensation, and/or other haptic effects. Corresponding visual effects can be provided.

<FIG> is a diagram showing how a user with one or more finger devices <NUM> may press fingers <NUM> against the exposed surfaces of an object <NUM> to measure the contours of object <NUM> in three dimensions. This allows system <NUM> to determine the shape of all sides of object <NUM>, including the far side of object <NUM> that might not be visible by a camera or other sensor in device <NUM> (e.g., in a scenario in which device <NUM> is a head-mounted device with a front-facing camera capturing real-world images).

Object <NUM> may be an inanimate object without circuitry (e.g., an object other than an electronic device such as a pencil, etc.) or may be an electronic device. In the example of <FIG>, object <NUM> is an electronic device with an input component such as component <NUM>' that may gather user input. Object <NUM> may be, for example, a touch screen display and component <NUM>' may be a touch sensor in the display. In another illustrative configuration, object <NUM> may be a computer mouse that gathers pointing input from a user and component <NUM>' may be a depressible button (switch) on the computer mouse. In yet additional scenarios, object <NUM> may be a cellular telephone, tablet computer, desktop computer, or keyboard, and component <NUM>' may be a touch screen, touch sensitive button, keyboard key, or other input device on object <NUM>.

During operation, system <NUM> may optionally use device <NUM> (e.g., a head-mounted display device) to display images that are overlaid on top of some or all of object <NUM>. Overlaid virtual content may, if desired, be used to transform a real-world object such as a stone or an ordinary household object into a control knob (e.g., a rotatable knob with haptic detents and/or other haptic features such as haptic-output button regions and with an appearance governed by the color, texture, and shape of the computer-generated image that is overlaid over the stone). In this way, electronic devices that are powered off, objects without circuitry, and/or other diverse objects in the user's surroundings can be used as input devices for controlling the operation of system <NUM>. As an example, virtual controllers can be constructed from pencils, erasers, baskets, cups, plates, furniture, shoes or other items of clothing, boxes or other enclosures, paper notebooks, and/or other items that can be touched, rotated, moved, and/or otherwise manipulated by a user.

In some arrangements, finger input from devices <NUM> may be gathered from a user and/or haptic output can be provided by devices 10when the user is wearing finger devices <NUM> during interactions with an object that is overlapped by virtual content. If desired, user input can be gathered using a gaze tracking system, depth sensors, cameras, and/or other sensors. Virtual content may, if desired, be overlaid over some portions of an object while leaving other portions uncovered. For example, a page in a notebook may contain handwritten notes. Virtual content such as computer-generated text annotations can be overlaid on a portion of the page adjacent to the notes. Pages in the notebook may have bar codes (e.g., QR codes) and/or other identifying information to help identify which pages are associated with corresponding annotation information (as an example).

In the illustrative configuration of <FIG>, object <NUM> is a pencil and does not contain any circuitry. A user wearing one or more finger devices <NUM> may rotate object <NUM> about longitudinal axis <NUM> of object <NUM>. The user may also move object <NUM> (e.g., the tip and/or end of object <NUM>) laterally, as indicated by lateral motions <NUM> of <FIG>. During movement of object <NUM>, finger devices <NUM> can gather information on the interactions between finger devices <NUM> and object <NUM> and can thereby be used in determining the location, orientation, and movement of object <NUM>.

Different interactions with object <NUM> may be used to control different corresponding functions in system <NUM>. For example, rotation of object <NUM> about axis <NUM> may adjust a volume or other analog property in system <NUM>, may rotate a virtual object, and/or may create other changes in the operation of system <NUM>. Lateral movement of object <NUM>, changes in the angular orientation of object <NUM> (e.g., tilting of object <NUM>), and/or other manipulations of object <NUM> can be used to produce other adjustments to the operation of system <NUM> (e.g., virtual object movement, adjusting of virtual object properties, adjusting hardware and/or software properties in system <NUM>, etc.). Object <NUM> may be used as a wand, sword, joystick, and/or item in a virtual world.

In some arrangements, some or all of object <NUM> may be overlaid with virtual content. As an example, icons, movable controls, and/or other interactive visual elements may be displayed over all of object <NUM> or in a select area of object such as illustrative area <NUM>. Area <NUM> may, as an example, include an array of virtual buttons that can be selected when a user presses finger <NUM> against a desired virtual button while wearing finger device <NUM>. A user may also provide input to area <NUM> or other parts of object <NUM> such a force input (e.g., different amounts of pressure on the sides of object <NUM> in area <NUM> as the user squeezes object <NUM> in the user's fingers), sliding input (e.g., a swipe gesture), etc. In some arrangements, the location of the tip of object <NUM> can be tracked by analyzing the positions of fingers <NUM> and/or by tracking images of object <NUM> gathered with an image sensor in device <NUM>. To help calibrate visual measurements of the shape and positions of the surfaces of object <NUM>, a user may touch the entire surface of object <NUM> while corresponding measurements are made with finger devices <NUM>, thereby determining the surface contours of object <NUM>, a user may cause object <NUM> to interact with other system components (e.g., by placing the tip of object <NUM> against a force sensor in device <NUM> while object <NUM> is being imaged with a camera, etc.).

If desired, a user can interact with a virtual object that has been scanned using a depth sensor, finger device, or other sensing arrangement (e.g. to change the color, size, shape, appearance, etc. of the object). In some arrangements, device(s) <NUM> may use pattern recognition to recognize object <NUM> and may take actions accordingly (e.g., by creating a magic wand skin for a recognized pencil).

In the configuration of <FIG>, system <NUM> includes multiple electronic devices. Device 24A in the example of <FIG> is an electronic device such as an internet-connected voice-controlled speaker or other device that does not include a display capable of displaying extensive text. Device 24B is a head-mounted device having a head-mountable support structure (housing) being worn by a user so that the user's eyes <NUM> can view content generated by device 24B and the real-world environment surrounding device 24A.

A user may view supplemental computer-generated content on device 24B such as text-based content while interacting with device 24A. A user may, for example, be wearing one or more finger devices <NUM>. The user may desire to view a list of available music tracks. In response to a finger command from finger devices <NUM> or other user input, device 24B may determine (directly or from wireless communications with device 24A) which tracks are available. The user may supply input commands with finger devices <NUM> or other input devices to navigate among virtual menu options and/or other options and thereby select a desired genre of music tracks to view. Computer-generated music track list <NUM> may then be displayed for the user by device 24B.

As an example, list <NUM> may be displayed adjacent to device 24A and/or overlapping some or all of device 24A. List <NUM> may contain selectable music tracks. A user may select a desired track by gesturing towards the track with finger device <NUM> (e.g., by placing the tip of the user's finger on a desired track for more than a predetermined amount of time, etc.), as illustrated by selected track <NUM> in <FIG>. In response to selection of track <NUM>, finger device <NUM> may provide haptic output (e.g., a click to confirm that track <NUM> has been selected), device 24B may take suitable action (e.g., by highlighting the track, by displaying information on the selected track, etc.), and device 24A may take suitable action (e.g., by playing the selected track in response to receiving the user finger input of the selection of track <NUM>). Wireless communications and/or wired communications between devices 24A, 24B, and finger devices <NUM> may be used in coordinating operation of the system. In this type of environment, device 24A may, if desired, serve as a host device and device 24B may serve as an input-output accessor coupled to device 24A (e.g., in addition to input-output finger device <NUM> being worn on the user's finger) and/or other networked equipment arrangements may be used.

If desired, selections of music tracks and other items in a computer-generated list or other virtual content may be made based on eye tracking information in addition to and/or instead of using finger input. For example, list <NUM> may be displayed in response to eye tracking information. For example, if a user's point-of-gaze is directed towards device 24A for more than a predetermined amount of time and/or is directed towards device 24A while a user makes a particular finger gesture using device(s) <NUM>, an interactive list such as list <NUM> can be automatically displayed to allow the user to make music track selections or other suitable selections.

Any of devices <NUM> may be controlled in this way. For example, device 24A of <FIG> may be a television, tablet computer, laptop computer, cellular telephone, and/or other adjustable equipment in a home or office (e.g., lights, thermostats, fans, audio equipment, shades, etc.). Gazed trackers in system <NUM> of <FIG> may be mounted on devices 24A and/or 24B. If desired, device 24A may include additional components (e.g., a display, etc.).

As shown in <FIG>, devices <NUM> may be used to interact with each other (e.g., to gather taps, swipes, pinch-to-zoom gesture input, etc.). If desired, devices <NUM> can measure the amount of finger pinching force generated as a user presses a pair of fingers <NUM> together. The amount of finger pinching force may be varied by a user dynamically to control the operation of system <NUM> in addition to other user finger input from devices <NUM>. As an example, a user may adjust the linewidth of a line drawn in a virtual three-dimensional space by a drawing application running on system <NUM> (e.g., on device <NUM> such as a head-mounted display or other device <NUM> with a display).

A graph of an illustrative three-dimensional line is shown in <FIG>. As shown in <FIG>, line <NUM> may have a linewidth W that varies as a function of distance along line <NUM>. The shape of line <NUM> may be determined by hand gestures (e.g., three-dimensional gestures through the air made by a user's hands). These motions, which may sometimes be referred to as three-dimensional finger gestures, may be gathered using inertial measurement units or other sensors in devices <NUM>. At the same time that a user is drawing line <NUM> on the three-dimensional workspace of <FIG>, the user may pinch fingers <NUM> together with a variable force. The amount of pinching force supplied by the user may be measured by finger devices <NUM> (e.g., using strain gauge circuitry) and may be used in adjusting the linewidth W dynamically. If, for example, the user desires to create a wide linewidth portion of line <NUM>, the user may momentarily increase the pinch force applied to finger devices <NUM>. In a pinch gesture, both devices <NUM> receive increased force measurements concurrently, so pinch gestures can be distinguished from gestures that involve forces applied to only a single finger device <NUM> such as a single-finger tap.

In general, any suitable attributes of a drawn line or other on-screen content can be adjusted using finger pinch gestures and/or other finger input (e.g., brush texture and other brush attributes, line color, object size, etc.). The use of pinch gesture input to adjust linewidth W is illustrative. Finger pinch gestures and other finger input from devices <NUM> may also be used in controlling zooming functions (e.g., to zoom in on an item of interest by performing a three-dimensional pinch-to-zoom operation, etc.), rotation operations (e.g., to rotate a three-dimensional object), moving an object along a selected axis, and/or other operations in system <NUM>.

<FIG> is a diagram of a mixed reality environment that may be presented to a user of device <NUM> while the user is interacting with device <NUM> using finger devices <NUM>. The surface of real-world tabletop <NUM> may be overlaid with computer-generated objects such as virtual horizontal surface <NUM>. Virtual vertical surface <NUM> may be displayed adjacent to virtual horizontal surface <NUM> (e.g., to form a virtual three-dimensional desktop). Surfaces <NUM> and <NUM> may, if desired, have colors, textures, and other visual attributes that help distinguish surfaces <NUM> and <NUM> from real-world objects. Icons, lists, menu options, text, still and/or moving images, and/or other virtual objects may be displayed on surfaces <NUM> and/or <NUM>. These virtual objects may be manipulated by a user using finger devices <NUM>.

In some configurations, surfaces <NUM> and <NUM> are not associated with any real-world objects (e.g., surfaces <NUM> and <NUM> may float). In other configurations, surface <NUM> and/or surface <NUM> may be aligned with the surface of a wall, table, or other non-electronic device and/or may be aligned with the surface of a display or other component in an electronic device. For example, surface <NUM> may be aligned with a table surface, so that movements of a user's finger and device <NUM> across the table may be used to drag-and-drop and otherwise move and manipulate virtual objects that are displayed visually on top of the table. At the same time, due to the physical presence of the table surface, the user's finger will contact the table (and finger device <NUM> will detect the contact and provide haptic feedback if desired) so that user selection of virtual objects on the table surface will feel natural to the user. If desired, trackpad or mouse input or other input to device <NUM> may be used in manipulating virtual objects on surfaces such as surface <NUM> (e.g., to move an object, etc.).

If desired, surface <NUM> may be aligned with a display in a desktop computer (see, e.g., illustrative optional structure <NUM> of <FIG>, which may be a desktop computer with a housing and a display in the housing), surface <NUM> may be aligned with the display in a tablet computer lying on tabletop <NUM> (see, e.g., illustrative optional structure <NUM> of <FIG>, which may be a tablet computer with a housing and a display in the housing), etc. These arrangements may be used to allow device <NUM> to gather input and provide haptic output while a user's finger is touching an object on the desktop computer display or on the display on the tablet computer. In the example of <FIG>, surface <NUM> is aligned with surface <NUM>. This is illustrative.

A user may interact with virtual content in the environment of <FIG> to control one or more associated devices <NUM> in system <NUM>. Consider, as an example, a virtual object such as virtual object <NUM>, which may be an email icon. A user who wishes to view an email inbox on a device <NUM> in system <NUM> may select object <NUM> by moving finger devices <NUM> inwardly in directions <NUM> towards object <NUM> with fingers <NUM>. Haptic output devices in devices <NUM> can supply appropriate haptic feedback as the user's fingers reach the surfaces of object <NUM>. After selecting the email icon, the user may move the email icon from surface <NUM> to surface <NUM> (e.g., using a drag and drop command, using a flick gesture, by picking up and throwing the icon, etc.). This directs the user's system (e.g., a computer and/or a head-mounted device <NUM> wirelessly communicating with the computer) to display email inbox <NUM> on surface <NUM>. After email inbox <NUM> has been displayed, a user can place a finger <NUM> on a listed email message subject line <NUM> in the email list of inbox <NUM> to select and open a desired email.

If desired, a user need not interact with virtual object <NUM> directly. For example, virtual pointers (e.g., a pair of virtual fingers) may appear on surface <NUM> and may be manipulated remotely by the user as the user's finger devices <NUM> are, for example, moved in directions <NUM> to grasp about object manipulation location <NUM> on real-world tabletop <NUM>. In this way, the virtual fingers may be directed to manipulate virtual object <NUM> (e.g., to move object <NUM> from surface <NUM> to surface <NUM> or to otherwise manipulate virtual objects in a virtual world being presented to the user).

When interacting with virtual objects such as virtual object <NUM> and list <NUM>, the user's point-of-gaze can be used as user input in addition to and/or instead of using finger input from finger devices <NUM>. As an example, the appearance of object <NUM> may be automatically enlarged or otherwise highlighted as a user's point-of-gaze is directed towards object <NUM>. The user may then grasp the object using finger devices <NUM> or other finger input may be supplied so that the object can be moved to surface <NUM>. In this way, virtual objects overlapping the real world (and/or real-time images of the real world) such as virtual object <NUM> may be manipulated using both point-of-gaze information and finger input from devices <NUM>.

An image sensor or other circuitry in the equipment of <FIG> may serve as a hand tracking system. For example, structure <NUM> of <FIG> may be a desktop computer with an image sensor and/or structure <NUM> of <FIG> may be a tablet computer or other electronic device (see, e.g., devices <NUM>) with an image sensor. The image sensor(s) can be used to capture images of a user's hand (e.g., a hand including finger <NUM>). By processing this image data, system <NUM> can track the position of the user's hand (and finger <NUM>). System <NUM> may, for example, be used to determine the distance of the user's hand from a display and to monitor horizontal and vertical movement of the hand relative to the display. By using a hand tracking system to determine the current position of the user's hand, the accuracy with which a computer or other equipment <NUM> can determine the position of finger device <NUM> and finger <NUM> can be enhanced.

For example, image-based hand tracking can be used alone or in combination with finger position data from device <NUM> to determine the location of device <NUM> and the user's finger relative to the device containing the image sensor and/or other circuitry of the hand tracking system (e.g., device <NUM>). Device <NUM> can then be used to gather force measurements or other user input indicative of touch events in which the user's finger contacts the surface of the display in structure <NUM> and/or structures <NUM> (e.g., to select virtual objects of interest, etc.). Haptic output can be provided in response to detected touch contact between the exposed surface of the user's finger and a display surface. In this way, coarse movement such as movement of device <NUM> due to hand movement can be accurately tracked (e.g., using an image based hand-tracking system or other hand tracker alone or in combination with finger and hand movement data from device <NUM>) while finger device <NUM> can use a force sensor or other senor to sensitively detect of touch events (e.g., taps on the surface of the display and other fine movements such as movements associated with touching and thereby selecting a virtual object so that a hand movement or other movement of finger <NUM> can be used to move the selected virtual object across the display). While manipulating displayed objects using hand position measurements from an image sensor or other sensor in the hand tracking system and/or using finger device position data, finger device <NUM> can use a haptic output device to provide associated haptic output such as haptic output associated with a touch event.

As this example demonstrates, a user may manipulate virtual objects (e.g., the user may select and move objects, may draw lines, etc.) by moving finger device <NUM> (whether the position of finger device <NUM> is detected using a hand tracking system in device <NUM>, position sensors in device <NUM>, and/or both hand tracking and position sensor measurements). For example, a user may select, drag, and drop virtual objects on the display of device <NUM>.

As the user interacts with system <NUM>, the user input from the sensor can be wirelessly conveyed from device <NUM> to device <NUM> in real time and control signals for the haptic output device in device <NUM> can be received from control circuitry within device <NUM>. The control signals for the haptic output device can originate within device <NUM> and/or can be provided to the haptic output device by the control circuitry within device <NUM> based on wirelessly received signals from device <NUM>.

<FIG> is a schematic diagram of an illustrative configuration for system <NUM>. In the example of <FIG>, system <NUM> includes a first device such as finger device <NUM> that communicates wirelessly over path <NUM> with an external electronic device (host) <NUM>. Device <NUM> may have storage and processing circuitry such as controller <NUM> and memory <NUM> and/or other control circuitry for controlling the operation of device <NUM>. During operation, the control circuitry of device <NUM> (see, e.g., control circuitry <NUM> of <FIG>) may use communications circuitry (see, e.g., communications circuitry <NUM> of <FIG>) to communicate with electronic device <NUM>. For example, controller <NUM> may use a wireless transceiver circuit such as Bluetooth® transceiver circuitry <NUM> and antenna <NUM> to communicate wirelessly with electronic device <NUM>.

Power for device <NUM> may be supplied using an energy storage device such as battery <NUM> or a supercapacitor (as examples). Battery <NUM> may be charged using power supplied through connectors in port <NUM> and/or power received wirelessly from a wireless power transmitter using wireless power receiving circuitry in port <NUM>. Charging circuitry <NUM> may supply received power to battery <NUM> to charge battery <NUM> and/or may direct received power to load circuitry in device <NUM>.

To enhance the ability of cameras and other tracking equipment to track the position and orientation of device <NUM>, controller <NUM> may use light-emitting diode drive circuitry <NUM> to turn on a set of light-emitting diodes <NUM> (e.g., a set of four light-emitting diodes, each located at a respective corner of the housing of device <NUM>). Controller <NUM> can supply haptic output to the finger of a user by driving piezoelectric actuator <NUM> using piezoelectric driver circuitry <NUM> or may use other haptic output devices <NUM> to provide haptic output.

Device <NUM> may gather input from sensors (see, e.g., sensors <NUM> of <FIG>) such as accelerometer and gyroscope <NUM> and compass <NUM>. This input may provide controller <NUM> with information on the orientation and movement of device <NUM>. Input from one or more strain gauges <NUM> and other sensor(s) <NUM> (e.g., force sensing resistors) may be gathered using analog-to-digital converter circuitry <NUM>. Information from the strain gauges <NUM> and other sensors may be indicative of forces on housing <NUM> and may therefore be used in measuring lateral movement of finger <NUM> (e.g., movement that causes housing <NUM> to bend so that this bending may be detected using a strain gauge, movement that causes finger <NUM> to press against force sensing resistors, etc.). The use of multiple strain gauges <NUM> and/or other sets of multiple sensors may help make system <NUM> more robust to failure. Input from multiple strain gauges <NUM> may also be used by controller <NUM> to determine touch location and/or touch direction. If desired, touch input from a capacitive touch sensor or a touch sensor based on force sensors, may be gathered. User input (e.g., finger input that is gathered using motion and orientation sensors and/or other sensors such as strain gauge circuitry and/or other force sensors) can be processed locally and/or may be transmitted wirelessly to device <NUM> for processing.

Bluetooth circuitry <NUM> may be compliant with a standard such as the Universal Serial Bus (USB) Human Interface Device (HID) standard. During operation, controller <NUM> can process raw sensor data to detect taps and other gestures and to detect pointing movements of device <NUM>. User input that has been detected can be transmitted wirelessly to electronic device <NUM>. With another illustrative arrangement, controller <NUM> sends measured sensor data to electronic device <NUM> for processing (e.g., for tap identification and other gesture processing). Other arrangements may be used, if desired.

When taps and other user input events are detected by controller <NUM>, controller <NUM> can supply haptic output to the user using driver <NUM> and actuator <NUM>. This haptic output may, as an example, be appropriate in situations in which it is desired to provide a user who has made a tap input or other input with device <NUM> with corresponding haptic feedback. In scenarios in which taps and other user input devices are processed using processing circuitry at device <NUM>, device <NUM> may wirelessly direct controller <NUM> to provide haptic output to the user's finger using driver <NUM> and piezoelectric actuator <NUM>. Haptic output waveforms (see, e.g., <FIG>) may be stored in memory <NUM>.

If desired, input may be gathered from multiple users using finger devices <NUM> and/or other equipment in system <NUM>. Output may also be provided to multiple users using finger devices <NUM> and/or other equipment in system <NUM>. In some arrangements, user input from finger devices <NUM> and/or other equipment in system <NUM> can be gathered from multiple users that are interacting with each other in a virtual environment (e.g., a game, a social application, or other application in an online environment, a local environment in which users interact via local wireless communications such as communications arrangements in which equipment in system <NUM> is interconnected by a wireless local area network and/or peer-to-peer wireless connection, or other collaborative environment). During these interactions between users, haptic output and/or other output may be provided to the users using finger devices <NUM> and/or other equipment in system <NUM>.

Control circuitry <NUM> in device(s) <NUM> and/or control circuitry <NUM> in device(s) <NUM> can take any suitable action in response to the input from multiple users. For example, if multiple users are playing a game and/or are interacting in a social application, control circuitry in system <NUM> may gather game and/or social application input from each of the users and, when appropriate, can provided haptic output or other output to one or more appropriate users (e.g., as a form of communication, as feedback related to in-game content, etc.). As another example, if two users are interacting socially (e.g., by shaking hands), control circuitry in system <NUM> may gather user input indicative of the shaking of hands from each of the users who is shaking hands and, in response, can take action such as providing haptic output to confirm to each of these users that business card information or other personal information has been exchanged wirelessly (e.g., a first type of haptic output indicating that contact information has been transmitted and/or a second type of haptic output indicating that contact information has been received). If desired, handshakes and other social interactions can take place online and haptic output or other output provided devices <NUM> and/or other equipment in system <NUM> accordingly (e.g., as feedback during an online handshake or other social interaction).

In some arrangements, devices <NUM> and/or other equipment in system <NUM> may be used by people with disabilities. Consider, as an example, a user with limited eyesight. This user may run an application on system <NUM> (e.g., on control circuitry <NUM> and/or <NUM>). The application may use sensors <NUM> and/or <NUM> to sense when the user is in the vicinity of an object (e.g., a physical element in the user's surroundings such as a wall, door, or step). Time-of-flight optical sensors, radio-frequency sensors, and/or other sensors for range detection (e.g., longrange proximity sensors) can be used to detect external objects as the user is interacting with the user's environment. In response to detecting external objects, corresponding haptic output, audio output, and/or other output can be provided to the user with finger device <NUM> and/or other equipment in system <NUM>. As an example, system <NUM> may be used to implement a virtual cane for the unsigned person of arbitrary length. As system <NUM> detects a step or other obstruction, haptic feedback or other output may inform the user of the presence of the step, so that the user can take appropriate action.

If desired, people with limited eyesight or other users may be provided with haptic output, audio output, and/or visual output using a head-mounted device (e.g., equipment <NUM> may include a head-mounted device) with or without accompanying output from finger device <NUM> in system <NUM>. As an obstruction or person outside of the user's field of view is detected, for example, system <NUM> may provide haptic vibrations to alert the user or can provide other output that indicates to the wearer of the head-mounted device and/or finger device that the obstruction or person is present. Users that have hearing impairment may also be provided with haptic output and/or other output in this way to indicate the presence of external objects, people in the presence of the user, etc..

In accordance with an embodiment, a system is provided that includes a display, control circuitry configured to receive finger input gathered by a finger device that is worn on a finger of a user while leaving a finger pad at the tip of the finger exposed, and a gaze tracker configured to produce point-of-gaze information associated with eyes of the user, the control circuitry is configured to use the display to display virtual content that is overlaid on real-world content and the control circuitry is configured to use the display to display the virtual content based on the finger input and the point-of-gaze information.

In accordance with another embodiment, the finger input includes strain gauge circuitry input gathered with strain gauge circuitry in the finger device and includes inertial measurement unit input gathered with an inertial measurement unit in the finger device, the virtual content is associated with a real-world object in the real-world content, and the control circuitry is configured to use the display to display a selectable item in the virtual content.

In accordance with another embodiment, the real-world object includes an electronic device and the control circuitry is configured to use the display to display the selectable item in a list.

In accordance with another embodiment, the finger device includes a haptic output device and the control circuitry is configured to gather the finger input as the finger device is being used to select the selectable item and is using the haptic output device to supply corresponding haptic output to the finger.

In accordance with another embodiment, the system includes a head-mounted support structure in which the display is mounted, the gaze tracker is mounted to the head-mounted support structure.

In accordance with another embodiment, the control circuitry is configured to receive additional finger input from an additional finger device and the control circuitry is configured to display the virtual content based on the finger input from the finger device and the additional finger input from the additional finger device.

In accordance with an embodiment, a head-mounted device operable with a finger device that is configured to gather finger input from a finger of a user is provided that includes control circuitry configured to receive the finger input from the finger device, head-mountable support structures, a display supported by the head-mountable support structures, and a gaze tracker configured to gather point-of-gaze information, the control circuitry is configured to use the display to display virtual content that is overlaid on real-world content and that is based on the point-of-gaze information and the finger input.

In accordance with another embodiment, the user has eyes, the point-of-gaze information is associated with the eyes of the user, and the control circuitry includes wireless communications circuitry configured to wirelessly receive the finger input from the finger device.

In accordance with another embodiment, the finger device includes an inertial measurement unit configured to gather the finger input as the finger interacts with the virtual content and the control circuitry is configured to use the display to display the virtual content based on the finger input gathered with the inertial measurement unit.

In accordance with another embodiment, the head-mountable support structures are configured to support the gaze tracker, the point-of-gaze information is gathered by the gaze tracker while the user is looking at an electronic device, and the virtual content includes a list associated with the electronic device.

In accordance with an embodiment, a finger device operable with a head-mounted device that is configured to display virtual content over a real-world object is provided that includes a housing configured to be coupled to a finger at a tip of the finger while leaving a finger pad at the tip of the finger exposed, a haptic output device coupled to the housing, sensor circuitry configured to gather finger position information as the finger interacts with the displayed virtual content, and control circuitry configured to provide haptic output to the finger using the haptic output device based on the finger position information.

In accordance with another embodiment, the sensor circuitry includes an inertial measurement unit configured to gather the position information while the user is interacting with the real-world object that is overlapped by the virtual content.

In accordance with another embodiment, the sensor circuitry includes strain gauge circuitry configured to gather a force measurement associated with a force generated by the finger as the finger contacts a surface of the real-world object while interacting with the displayed virtual content.

In accordance with another embodiment, the sensor circuitry includes an ultrasonic sensor.

In accordance with another embodiment, the sensor circuitry includes an image sensor.

In accordance with another embodiment, the sensor circuitry includes a depth sensor.

In accordance with another embodiment, the control circuitry is configured to provide the haptic output with the haptic output device while the finger is selecting an item in an interactive list in the virtual content.

In accordance with another embodiment, the real-world object does not contain circuitry and the sensor circuitry is configured to gather the finger position information as the finger interacts with the displayed virtual content while simultaneously moving the real object.

In accordance with another embodiment, the control circuitry is configured to provide the finger with different textures in different areas of the real-world object by providing different haptic output to the finger in the different areas using the haptic output device based on the finger position information.

In accordance with another embodiment, the virtual content includes a virtual object and the control circuitry is configured to move the virtual object in response to the finger position information.

In accordance with an embodiment, a finger device configured to be worn on a finger of a user is provided that includes a housing configured to be coupled to a tip of the finger while leaving a finger pad at the tip of the finger exposed, a haptic output device coupled to the housing, a touch sensor configured to gather touch input along an exterior surface of the housing, and control circuitry configured to provide haptic output to the finger using the haptic output device based on the gathered touch input.

In accordance with another embodiment, the touch sensor includes an array of capacitive touch sensor electrodes extending along the housing, the touch sensor is configured to gather the touch input from an additional finger touching the touch sensor, and the control circuitry is configured to create haptic detents by providing the haptic output as the additional finger moves along the array of capacitive touch sensor electrodes.

In accordance with another embodiment, the housing is configured to receive the finger without covering a finger pad at a tip of the finger and the touch sensor has an array of sensor elements that extend along the housing.

In accordance with another embodiment, the sensor elements include capacitive touch sensor electrodes.

In accordance with another embodiment, the finger device includes a sensor coupled to the housing that gathers information on interactions of the finger with external objects.

In accordance with another embodiment, the sensor coupled to the housing includes a sensor selected from the group consisting of an inertial measurement unit and a strain gauge.

In accordance with another embodiment, the sensor coupled to the housing includes an image sensor.

In accordance with another embodiment, the sensor coupled to the housing includes a depth sensor.

In accordance with another embodiment, the sensor coupled to the housing includes a strain gauge coupled to an elongated arm in the housing.

In accordance with another embodiment, the finger device includes sensor elements configured to gather touch input from an area on the finger that is adjacent to the housing and that is not overlapped by the housing.

In accordance with another embodiment, the finger device includes a force sensor configured to gather finger pinch input from the finger as the finger presses against another finger.

In accordance with an embodiment, a finger device operable with a wireless electronic device and configured to be worn on a first finger of a user while receiving touch input from a second finger of the user is provided that includes a housing configured to be coupled to the first finger, a touch sensor configured to gather touch input along an exterior surface of the housing from the second finger, and control circuitry configured to wirelessly transmit the gathered touch input to the wireless electronic device.

In accordance with another embodiment, the finger device includes a haptic output device configured to provide haptic output.

In accordance with another embodiment, the control circuitry is configured to use the haptic output device to create detents as the second finger moves along the touch sensor.

In accordance with another embodiment, the finger device includes a sensor configured to gather finger input associated with the first finger.

In accordance with another embodiment, the control circuitry is configured to use the haptic output device to provide the first finger with haptic output based on the finger input.

In accordance with another embodiment, the sensor includes an accelerometer and the finger input includes finger motion sensed with the accelerometer.

In accordance with another embodiment, the sensor includes an inertial measurement unit.

In accordance with another embodiment, the sensor includes a force sensor.

In accordance with another embodiment, the sensor includes an ultrasonic sensor.

In accordance with another embodiment, the sensor includes an optical sensor.

In accordance with another embodiment, the housing is configured to be coupled to a tip of the first finger while leaving a finger pad at the tip of the first finger exposed.

In accordance with another embodiment, the housing includes a U-shaped housing having first and second sides coupled by a top and the touch sensor is configured to gather touch input on the first side, the second side, and the top.

In accordance with another embodiment, the wireless electronic device includes a head-mounted device configured to display virtual content that overlaps the exterior surface as the touch sensor gathers the touch input.

In accordance with another embodiment, the housing has a bendable arm with a strain gauge and the touch sensor overlaps the bendable arm.

In accordance with an embodiment, a finger device operable with a wireless electronic device and configured to be worn on a first finger of a user while receiving touch input from a second finger of the user is provided that includes a housing configured to be coupled to the first finger, a sensor configured to gather finger input from the second finger a region on the first finger that is not overlapped by the housing, and control circuitry configured to wirelessly transmit the gathered finger input to the wireless electronic device.

In accordance with another embodiment, the finger device includes orthogonal first, second, and third directional sensors, the control circuitry is configured to gather information on an orientation of the first finger from the first, second, and third directional sensors.

In accordance with another embodiment, the first, second, and third sensors include radio-frequency sensors.

In accordance with another embodiment, the housing has a portion that protrudes in front of a finger tip portion of the first finger and includes sensor circuitry configured to gather information on movement of the first finger.

In accordance with an embodiment, a finger device operable with a computer having a display is provided that includes a housing configured to be worn on a finger of a user, a haptic output device, a sensor configured to gather user input as the finger of the user touches the display, wireless communications circuitry configured to transmit the user input to the computer, and control circuitry configured to use the haptic output device to provide haptic output to the finger of the user as an exposed surface of the finger touches the display.

In accordance with another embodiment, the computer includes a tablet computer, the display is configured to display an object, and the control circuitry is configured to use the sensor to gather the user input while the finger touches the object.

In accordance with another embodiment, the finger device includes a touch sensor configured to gather touch input along an exterior surface of the housing, the control circuitry is configured to provide haptic output to the finger using the haptic output device based on the gathered touch input.

In accordance with another embodiment, the computer includes a desktop computer, the display is configured to display an object, and the control circuitry is configured to use the sensor to gather the user input while the finger touches the object.

In accordance with an embodiment, an electronic device operable with a finger device that is configured to be worn on a finger of a hand of a user and that has a sensor configured to gather user input, is provided that includes a display, a hand-tracking system configured to measure movement of the hand of the user, wireless communications circuitry configured to receive user input gathered with the sensor in the finger device, and control circuitry configured to move an object on the display in response to the measured movement of the hand and in response to the user input.

In accordance with an embodiment. a finger device operable with a head-mounted device that is configured to display virtual content over a real-world object, the finger device is provided that includes a housing configured to be coupled to a finger at a tip of the finger while leaving a finger pad at the tip of the finger exposed, a haptic output device coupled to the housing, sensor circuitry configured to gather finger position information as the finger interacts with the displayed virtual content, and control circuitry configured to provide haptic output to the finger using the haptic output device based on the finger position information.

In accordance with another embodiment, control circuitry is configured to provide the haptic output with the haptic output device while the finger is selecting an item in an interactive list in the virtual content.

In accordance with another embodiment, the finger device defined includes sensor elements configured to gather touch input from an area on the finger that is adjacent to the housing and that is not overlapped by the housing.

In accordance with an embodiment, a finger device operable with a wireless electronic device and configured to be worn on a first finger of a user while receiving touch input from a second finger of the user is provided that includes a housing configured to be coupled to the first finger, a sensor configured to gather finger input from the second finger within a region on the first finger that is not overlapped by the housing, and control circuitry configured to wirelessly transmit the gathered finger input to the wireless electronic device.

In accordance with an embodiment, a finger device operable with a computer having a display, the finger device is provided that includes a housing configured to be worn on a finger of a user, a haptic output device, a sensor configured to gather user input as the finger of the user touches the display, wireless communications circuitry configured to transmit the user input to the computer, and control circuitry configured to use the haptic output device to provide haptic output to the finger of the user as an exposed surface of the finger touches the display.

Claim 1:
A system, comprising:
a display;
control circuitry (<NUM>) configured to receive finger input gathered by a finger device (<NUM>) that is worn on a finger (<NUM>) of a user while leaving a finger pad at the tip of the finger exposed, wherein the finger device (<NUM>) has a U-shaped housing having first and second sides (44SW) coupled by a top portion (44T);
a gaze tracker (<NUM>) configured to produce point-of-gaze information associated with eyes of the user, characterised in that the control circuitry (<NUM>) is configured to use the display to display virtual content that is overlaid on a real-world object (<NUM>) and wherein the control circuitry (<NUM>) is configured to use the display to display the virtual content based on the finger input and the point-of-gaze information, and sensor circuitry configured to gather finger position information as the finger interacts with the displayed virtual content, and to determine a movement of the real-world object (<NUM>) based on the finger position information.