Computer systems with finger devices for sampling object attributes

A system may include electronic devices. The electronic devices may include finger devices configured to be worn on fingers of users and may include other electronic devices. The electronic devices may include sensors. A sensor may measure information on real-world-object physical attributes such as surface contours and textures associated with real-world objects. The measured physical attributes may be used to recreate a sampled physical object for a user in a computer-generated reality environment. During presentation of computer-generated content, visual content, audio content, haptic content, and/or other content may be presented that is based on measured visual, audio, haptic, and/or other physical attributes of a real-world object. Content may be presented using a head-mounted device, haptic devices and other output devices in finger devices, and/or other output devices.

FIELD

This relates generally to electronic systems, and, more particularly, to systems with electronic devices such as finger-mounted electronic devices.

BACKGROUND

Electronic devices such as computers can be controlled using computer mice and other input accessories. In computer-generated reality systems, force-feedback gloves can be used to control virtual objects. Cellular telephones may have touch screen displays and vibrators that are used to create haptic feedback in response to touch input.

Devices such as these may not be convenient for a user, may be cumbersome or uncomfortable, or may provide unrealistic output.

SUMMARY

A system may include electronic devices. The electronic devices may include finger devices configured to be worn on fingers of users and may include head mounted devices and other electronic devices. The electronic devices may include sensors. As a user interacts with a real-world object in the environment surrounding the user, one or more sensors may be sued to measure information on real-world-object physical attributes associated with the real-world object. The physical attributes that are measured may include attributes such as a surface contour, a texture, an object color or other visual attribute, a temperature, acoustic attributes, force-versus-distance characteristics, weight, and/or other physical attributes.

The measured physical attributes may be used to recreate the physical behavior of a portion of a sampled physical object for a user. For example, a sampled texture may be overlaid on a part of an object being presented in a computer-generated reality environment. During presentation of computer-generated content, visual content, audio content, haptic content, and/or other content may be presented that includes measured visual attributes, audio attributes, haptic attributes, and/or other sampled physical attributes of a real-world object. Content may be presented using a head-mounted device, haptic devices and other output devices in finger devices, and/or other output devices in the electronic devices.

DETAILED DESCRIPTION

Electronic devices may be used to gather user input and to provide a user with output. For example, an electronic device may capture information on the physical attributes of real-world objects in an environment surrounding a user. Position sensors such as inertial measurement units and other sensors that can detect motion and location, force sensors, image sensors, and other sensors may be used in gathering measurements of real-world object physical attributes as a user interacts with the physical world. Samples of textures, visual patterns, measured objects shapes, and other real-world information can be gathered and stored. When using playback equipment such as a finger-mounted device, head-mounted device, and/or other electronic equipment, sampled real-world attributes can be provided to a user. Sampled real-world-object attributes may, for example, be provided to a user using haptic output devices, audio and visual output devices, and/or other output devices while a user interacts with real-world and computer-generated content.

An electronic system that allows a user to gather measurements of real-world-object physical attributes and that provides sampled attributes to a user may include electronic devices such as cellular telephones and computers. If desired, the electronic system may include wearable electronic devices that are configured to be mounted on the body of a user. For example, the electronic system may include devices that that are configured to be worn on one or more of a user's fingers. These devices, which may sometimes be referred to as finger devices or finger-mounted devices, may be used to gather input and supply output. A finger device may, as an example, include sensors that measure object surface shape and responses to applied pressure. The visual appearance and other physical attributes of real-world objects can also be measured using sensor circuitry in a finger device.

Wearable electronic devices such as head-mounted devices may also be used in measuring physical attributes of real-world objects. Sampled real-world object physical attributes can be played back to a user using wearable electronic devices as a user interacts with real and/or virtual objects. For example, a sampled real-world texture may be recreated using a haptic output component in a finger device as a user touches a real world object. Visual content such as sampled real-world visible attributes can also be provided to the user. For example, a display in a head-mounted device may be used to overlay a previously sampled surface appearance of a real-world object onto a different real-world object. Haptic output from a finger device or other equipment and visual output from a head-mounted device may, if desired, be provided to a user simultaneously and in coordination with each other as a user is interacting with real-world and virtual content.

If desired, other input may be gathered using one or more wearable electronic devices or other electronic devices and other output may be provided to a user while the user is using the electronic system. The use of a finger device to gather input and to provide corresponding haptic output and the use of a head-mounted display to display visual content for a user is illustrative.

During sampling a user may measure real-world-object physical attributes using one or more finger devices, head-mounted devices, and/or other electronic devices. These devices may also gather user input during operation of the system. During playback operations, an electrical system may provide the user with computer-generated content (sometimes referred to as virtual content) based on the sampled real-world object physical attributes and/or may provide the user with other computer-generated content. User input may be used in moving virtual objects and otherwise controlling system operations. If desired, the user may receive output from the electrical system while interacting with real-world objects. For example, haptic output corresponding to a previously sampled real-world object texture may be provided to a user while the user is touching a real-world object. In this way, a computer-generated version of a sampled real-world texture may be overlaid on a texture on a real-world surface and/or may replace a real-world texture associated with a real-world object that a user is touching. In some configurations, haptic output and other output may be supplied while a user's fingers are moving through the air without contacting any real-world objects.

Haptic output, visual output, audio output, and/or other output (e.g., heat, etc.) may be supplied by one or more devices in the electronic system. One or more devices may also be used in gathering user input. In some configurations, a user may use finger devices when using the electronic system to produce a computer-generated reality environment. This system may include one or more electronic devices that produce visual and audio output such as head-mounted equipment. Head-mounted devices may include glasses, goggles, a helmet, or other devices with displays and, if desired, speakers. During operation, finger devices may gather user input such as information on interactions between the finger device(s) and the surrounding environment (e.g., interactions between a user's fingers and the environment, including finger motions and other interactions associated with virtual content displayed for a user). The user input may be used in controlling visual output on the display. Corresponding haptic output may be provided during operation. This haptic output may include previously sampled real-world object physical attributes such as object shape, texture, response to pressure, etc. and may be provided to the user's fingers using the finger devices. Haptic output may be used, for example, to provide the fingers of a user with a desired texture sensation as a user is touching a real object or as a user is touching a virtual object. Haptic output can also be used to create detents and other haptic effects, to create force feedback that makes virtual objects that are hovering in space appear real to the touch.

Finger devices can be worn on any or all of a user's fingers (e.g., the index finger, the index finger and thumb, three of a user's fingers on one of the user's hands, some or all fingers on both hands, etc.). To enhance the sensitivity of a user's touch as the user interacts with surrounding objects, finger devices may have inverted U shapes or other configurations that allow the finger devices to be worn over the top and sides of a user's finger tips while leaving the user's finger pads exposed. This allows a user to touch objects with the finger pad portions of the user's fingers during use. If desired, finger devices may be worn over knuckles on a user's finger, between knuckles, and/or on other portions of a user's finger. The use of finger devices on a user's finger tips is sometimes described herein as an example.

Users can use the finger devices to interact with any suitable electronic equipment. For example, a user may use one or more finger devices to interact with an electronic system that supplies a computer-generated-reality environment. This equipment may include a head-mounted device with a display and, if desired, an associated host system such as a computer and/or cloud computing equipment that is accessed by the head-mounted display and/or cloud computing equipment that is accessed using the host system. Computer-generated-reality equipment may also include devices such as a tablet computer, cellular telephone, watch, ear buds, stylus, or other accessory, and/or other electronic equipment. In some systems, finger devices may be augmented or replaced by other electronic devices such as touch and/or force sensitive haptic-output gloves (sometimes referred to as computer-generated reality controller gloves), joysticks, touch pads, styluses, keyboards, computer mice, and/or other input-output devices.

FIG. 1is a schematic diagram of an illustrative system of the type that may include one or more finger devices and/or other input-output devices for sampling real-world object physical attributes. As shown inFIG. 1, system8may include electronic device(s) such as finger device(s)10and other electronic device(s)24. Each finger device10may be worn on a finger of a user's hand. Additional electronic devices in system8such as devices24may include devices such as a laptop computer, a computer monitor containing an embedded computer, a tablet computer, a desktop computer (e.g., a display on a stand with an integrated computer processor and other computer circuitry), a cellular telephone, a media player, or other handheld or portable electronic device, a smaller device such as a wristwatch device, a pendant device, a headphone or earpiece device, a head-mounted device such as glasses, goggles, a helmet, or other equipment worn on a user's head, or other wearable or miniature device, a television, a computer display that does not contain an embedded computer, a gaming device, a remote control, a navigation device, an embedded system such as a system in which equipment is mounted in a kiosk, in an automobile, airplane, or other vehicle, a removable external case for electronic equipment, a strap, a wrist band or head band, a removable cover for a device, a case or bag that has straps or that has other structures to receive and carry electronic equipment and other items, a necklace or arm band, a wallet, sleeve, pocket, or other structure into which electronic equipment or other items may be inserted, part of a chair, sofa, or other seating (e.g., cushions or other seating structures), part of an item of clothing or other wearable item (e.g., a hat, belt, wrist band, headband, sock, glove, shirt, pants, etc.), a mouse, trackpad, stylus, ear buds, or other accessories, or equipment that implements the functionality of two or more of these devices.

Devices24may, if desired, include cloud-based computing equipment (e.g., one or more computers that are accessed over the Internet or other wide area network and/or over local area networks). Network communications paths may be wired and/or wireless. Cloud-based computers, which may sometimes be referred to as servers or online computers, may be used to store libraries of sampled real-world-object physical attributes and other shared and/or user-generated content. For example, a sampled texture from a given user may be uploaded to an online computer and subsequently downloaded for use by the computer-generated reality system of the user or another user.

In some arrangements, a single device24(e.g., a head-mounted device) may be used with one or more devices10. In other arrangements, multiple devices24(e.g., a head-mounted device and an associated host computer or a head-mounted device, host computer, and online computer) may be used in system8with one or more devices10. In yet other configurations, system8includes only one or more devices10(e.g., a head-mounted device, a cellular telephone, or a finger-mounted device, etc.). Configurations in which system10includes one or more devices10and one or more devices24may sometimes be described herein as an example.

With one illustrative configuration, device10is a finger-mounted device having a finger-mounted housing (finger device housing) with a U-shaped body that grasps a user's finger or a finger-mounted housing with other shapes configured to rest against a user's finger and device(s)24is a cellular telephone, tablet computer, laptop computer, wristwatch device, head-mounted device, a device with a speaker, or other electronic device (e.g., a device with a display, audio components, and/or other output components). A finger device with a U-shaped housing may have opposing left and right sides that are configured to receive a user's finger and a top housing portion that couples the left and right sides and that overlaps the user's fingernail.

Devices10and24may include control circuitry12and26. Control circuitry12and26may include storage and processing circuitry for supporting the operation of system8. The storage and processing circuitry may include storage such as nonvolatile memory (e.g., flash memory or other electrically-programmable-read-only memory configured to form a solid state drive), volatile memory (e.g., static or dynamic random-access-memory), etc. Processing circuitry in control circuitry12and26may be used to gather input from sensors and other input devices and may be used to control output devices. The processing circuitry may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors and other wireless communications circuits, power management units, audio chips, application specific integrated circuits, etc.

To support communications between devices10and24and/or to support communications between equipment in system8and external electronic equipment, control circuitry12may communicate using communications circuitry14and/or control circuitry26may communicate using communications circuitry28. Circuitry14and/or28may include antennas, radio-frequency transceiver circuitry, and other wireless communications circuitry and/or wired communications circuitry. Circuitry14and/or26, which may sometimes be referred to as control circuitry and/or control and communications circuitry, may, for example, support bidirectional wireless communications between devices10and24over wireless link38(e.g., a wireless local area network link, a near-field communications link, or other suitable wired or wireless communications link (e.g., a Bluetooth® link, a WiFi® link, a 60 GHz link or other millimeter wave link, etc.). Devices10and24may also include power circuits for transmitting and/or receiving wired and/or wireless power and may include batteries. In configurations in which wireless power transfer is supported between devices10and24, in-band wireless communications may be supported using inductive power transfer coils (as an example).

Devices10and24may include input-output devices such as devices16and30. Input-output devices16and/or30may be used in gathering user input, in gathering information on the environment surrounding the user, and/or in providing a user with output. Devices16may include sensors18and devices24may include sensors32. Sensors18and/or32may include force sensors (e.g., strain gauges, capacitive force sensors, resistive force sensors, etc.), audio sensors such as microphones, touch and/or proximity sensors such as capacitive sensors, optical sensors such as optical sensors that emit and detect light, ultrasonic sensors, and/or other touch sensors and/or proximity sensors, monochromatic and color ambient light sensors, image sensors, sensors for detecting position, orientation, and/or motion (e.g., accelerometers, magnetic sensors such as compass sensors, gyroscopes, and/or inertial measurement units that contain some or all of these sensors), muscle activity sensors (EMG) for detecting finger actions, radio-frequency sensors, depth sensors (e.g., three-dimensional optical sensors such as structured light sensors configured to project dots of infrared light onto three-dimensional surfaces of real-world objects and sense three-dimensional shapes by capturing images of the dots using an infrared image sensor and/or optical depth sensors based on stereo imaging devices), optical sensors such as self-mixing sensors and light detection and ranging (lidar) sensors that gather time-of-flight measurements, optical sensors such as visual odometry sensors that gather position and/or orientation information using images gathered with digital image sensors in cameras, gaze tracking sensors, visible light and/or infrared cameras having digital image sensors, humidity sensors, moisture sensors, sensors that detect finger bending and other user movements, and/or other sensors. In some arrangements, devices10and/or24may use sensors18and/or32and/or other input-output devices16and/or30to gather user input (e.g., buttons may be used to gather button press input, touch sensors overlapping displays can be used for gathering user touch screen input, touch pads may be used in gathering touch input, microphones may be used for gathering audio input, accelerometers may be used in monitoring when a finger contacts an input surface and may therefore be used to gather finger press input, etc.). If desired, device10and/or device24may include rotating buttons (e.g., a crown mechanism on a watch or finger device or other suitable rotary button that rotates and that optionally can be depressed to select items of interest). Alphanumeric keys and/or other buttons may be included in devices16and/or30.

Devices16and/or30may include haptic output devices20and/or34. Haptic output devices20and/or34can produce motion that is sensed by the user (e.g., through the user's fingertips). Haptic output devices20and/or34may include actuators such as electromagnetic actuators, motors, piezoelectric actuators, electroactive polymer actuators, vibrators, linear actuators, rotational actuators, actuators that bend bendable members (e.g., actuators that apply bending force across one or more joints in a finger), actuator devices that create and/or control repulsive and/or attractive forces between devices10and/or24(e.g., components for creating electrostatic repulsion and/or attraction such as electrodes, components for producing ultrasonic output such as ultrasonic transducers, components for producing magnetic interactions such as electromagnets for producing direct-current and/or alternating-current magnetic fields, permanent magnets, magnetic materials such as iron or ferrite, and/or other circuitry for producing repulsive and/or attractive forces between devices10and/or24). In some situations, actuators for creating forces in device10may be used in squeezing a user's finger and/or otherwise directly interacting with a user's finger pulp. In other situations, these components may be used to interact with each other (e.g., by creating a dynamically adjustable electromagnetic repulsion and/or attraction force between a pair of devices10and/or between device(s)10and device(s)24using electromagnets).

If desired, input-output devices16and/or30may include other devices22and/or36such as displays. The displays may include, for example, a liquid crystal display, an organic light-emitting diode display, or other display with an array of pixels on which an image is displayed for a user. For example, device24and/or device10may include a display mounted on an exterior device face and/or in a viewable interior location that displays an image for a user. If desired, input-output devices16and/or30may include projector displays (projectors) that project images onto table tops or other external surfaces in the vicinity of the user. In this type of arrangement, a user may, as an example, view content that is being projected by a projector in device10and/or by a projector in device24onto the external surface while using device10to interact with the projected image. Input-output devices16and/or30may, if desired, include other input-output components such status indicator lights (e.g., a light-emitting diode in device10and/or24that serves as a power indicator, and other light-based output devices), speakers and other audio output devices, electromagnets, permanent magnets, structures formed from magnetic material (e.g., iron bars or other ferromagnetic members that are attracted to magnets such as electromagnets and/or permanent magnets), batteries, etc. Devices10and/or24may also include power transmitting and/or receiving circuits configured to transmit and/or receive wired and/or wireless power signals. If desired, other devices22and36may include heating and/or cooling elements such as resistive heaters, thermoelectric cooling elements based on the Peltier effect, or other adjustable temperature sources.

FIG. 2is a top view of a user's finger (finger40) and an illustrative finger-mounted device10. As shown inFIG. 2, device10may be formed from a finger-mounted unit that is mounted on or near the tip of finger40(e.g., partly or completely overlapping fingernail42). If desired, device10may be worn elsewhere on a user's fingers such as over a knuckle, between knuckles, etc. Configurations in which a device such as device10is worn between fingers40may also be used. As shown by illustrative optional portions10E and10B, device10may be configured to overlap one or more joints in finger40. This allows finger joint bending to be monitored using a bend sensor (e.g., a bend sensor in portion10B of device10that measures movement of portion10E relative to the remainder of device10as a user's finger bends). Portion10B may also include haptic output devices that apply bending force to finger40(e.g., an electromagnetic actuator that applies forces to finger40that tend to bend finger40about the joint overlapped by device10).

A user may wear one or more of devices10simultaneously. For example, a user may wear a single one of devices10on the user's ring finger or index finger. As another example, a user may wear a first device10on the user's thumb, a second device10on the user's index finger, and an optional third device10on the user's middle finger. Arrangements in which devices10are worn on other fingers and/or all fingers of one or both hands of a user may also be used.

Control circuitry12(and, if desired, communications circuitry14and/or input-output devices16) may be contained entirely within device10(e.g., in a housing for a fingertip-mounted unit) and/or may include circuitry that is coupled to a fingertip structure (e.g., by wires from an associated wrist band, glove, fingerless glove, etc.). Configurations in which devices10have bodies that are mounted on individual user fingertips are sometimes described herein as an example.

FIG. 3is a cross-sectional side view of an illustrative finger device (finger-mounted device)10showing illustrative mounting locations46for electrical components (e.g., control circuitry12, communications circuitry14, and/or input-output devices16) within and/or on the surface(s) of finger device housing44. These components may, if desired, be incorporated into other portions of housing44.

As shown inFIG. 3, housing44may have a U shape (e.g., housing44may be a U-shaped housing structure that faces downwardly and covers the upper surface of the tip of user finger40and fingernail42). During operation, a user may press against structures such as structure50(e.g., a real-world object). As the bottom of finger40(e.g., finger pulp40P) presses against surface48of structure50, the user's finger may compress and force portions of the finger outwardly against the sidewall portions of housing44(e.g., for sensing by force sensors or other sensors mounted to the side portions of housing44). Lateral movement of finger40in the X-Y plane may also be sensed using force sensors or other sensors on the sidewalls of housing44or other portions of housing44(e.g., because lateral movement will tend to press portions of finger40against some sensors more than others and/or will create shear forces that are measured by force sensors that are configured to sense shear forces).

Ultrasonic sensors, optical sensors, inertial measurement units, strain gauges and other force sensors, radio-frequency sensors, and/or other sensors may be used in gathering sensor measurements indicative of the activities of finger40. If desired, these sensors may also be used in mapping the contours of three-dimensional objects (e.g., by time-of-flight measurements and/or other measurements). For example, an ultrasonic sensor such as a two-dimensional image sensor or an ultrasonic sensor with a single ultrasonic transducer element may emit free-space ultrasonic sound signals that are received and processed after reflecting off of external objects. This allows a three-dimensional ultrasonic map to be generated indicating the shapes and locations of the external objects.

In some configurations, finger activity information (position, movement, orientation, etc.) may be gathered using sensors that are mounted in external electronic equipment (e.g., in a computer or other desktop device, in a head-mounted device or other wearable device, and/or in other electronic device24that is separate from device10). For example, optical sensors such as images sensors that are separate from devices10may be used in monitoring devices10to determine their position, movement, and/or orientation. If desired, devices10may include passive and/or active optical registration features to assist an image sensor in device24in tracking the position, orientation, and/or motion of device10. For example, devices10may include light-emitting devices such as light-emitting diodes and/or lasers. The light-emitting devices may be arranged in an asymmetric pattern on housing44and may emit light that is detected by an image sensor, depth sensor, and/or other light-based tracking sensor circuitry in device24. By processing the received patterned of emitted light, device24can determine the position, orientation, and/or motion of device10. The positions (e.g., surface contours) of surfaces may be detected by measuring the position of device10when device10experiences a jolt, experiences a touch sensor touch event, or experiences a spike in other appropriate sensor output due to contact with the surface. The jolt may create a spike in an accelerometer output, a capacitive force sensor output, a strain gauge output, or other touch and/or force sensing circuit output in device10. Surface contours may also be measured optically, using radio-frequency signals, using acoustic signals, etc.

If desired, finger device tracking can be performed that involves extrapolating from a known body part orientation (e.g., a finger orientation) to produce orientation information on other body parts (e.g., wrist and/or arm orientation estimated using inverse kinematics). Visual odometry sensors may, if desired, be included in devices10. These sensors may include image sensors that gather frames of image data of the surroundings of devices10and may be used in measuring position, orientation, and/or motion from the frame of image data. Lidar, ultrasonic sensors oriented in multiple directions, radio-frequency tracking sensors, and/or other finger device tracking arrangements may be used, if desired.

In some arrangements, user input for controlling system8can include both user finger input and other user input (e.g., user eye gaze input, user voice input, etc.). For example, gaze tracking information such as a user's point-of-gaze measured with a gaze tracker can be fused with finger input when controlling device10and/or devices24in system8. The finger input may include information on finger orientation, position, and/or motion and may include information on how forcefully a finger is pressing against surfaces (e.g., force information). By monitoring finger position while also measuring touch sensor output, force sensor output, and/or output from other sensors, information may be gathered on the surface shapes of real-world objects and other real-world physical attributes. For example, if a user touches a real-world object, device10can detect that the user's finger has contacted the real-world object and can detect the location of the contact event, thereby mapping out the surface shape of the real-world object. In this way, surface textures, the response of an object to applied force, global surface shapes, object temperature, and other real-world object physical attributes can be obtained.

The sensors in device10may, for example, measure how forcefully a user is moving device10(and finger40) against a real-world object surface such as surface48(e.g., in a direction parallel to the surface normal n of surface48such as the −Z direction ofFIG. 3) and/or how forcefully a user is moving device10(and finger40) within the X-Y plane, tangential to surface48. The direction of movement of device10in the X-Y plane and/or in the Z direction can also be measured by the force sensors and/or other sensors18at locations46.

Structure50may be a portion of a housing of device24, may be a portion of another device10(e.g., another housing44), may be a portion of a user's finger40or other body part, may be a surface of a real-world object such as a table, a movable real-world object such as a bottle or pen, or other inanimate object external to device10, and/or may be any other real-world object that the user can contact with finger40while moving finger40in a desired direction with a desired force and/or any other structure that the user can measure using sensors in device10. Because finger motions can be sensed by device10, device(s)10can also be used to gather pointing input (e.g., input moving a cursor or other virtual object on a display such as a display in devices36), can be used to gather tap input, swipe input, pinch-to-zoom input (e.g., when a pair of devices10is used), or other gesture input (e.g., finger gestures, hand gestures, arm motions, etc.), and/or can be used to gather other user input.

System8may include an optical sensor such as a gaze detection sensor (sometimes referred to as a gaze detector, gaze tracker, gaze tracking system, or eye monitoring system). A gaze tracking system for system8may, for example, include image sensors, light sources, and/or other equipment that is used in monitoring the eyes of a user. This system may include one or more visible and/or infrared cameras that face a viewer's eyes and capture images of the viewer's (user's) eyes. During operation of system8, control circuitry in system8(e.g., control circuitry coupled to a housing in device24) may use the gaze tracking system to track a viewer's gaze. Cameras and/or other sensors in device24may, for example, determine the location of a user's eyes (e.g., the centers of the user's pupils) and may determine the direction in which the user's eyes are oriented.

The orientation of the user's gaze may be used to determine the location in a computer-generated environment in which a user's eyes are directed (sometimes referred to as the user's point-of-gaze). If desired, device24and/or other equipment in system8may use gaze tracking information such as information on the user's point-of-gaze in determining which actions to take in system8. For example, a gaze tracking system may determine that a user's point-of-gaze is directed towards a first object and not a second object and may respond by assuming that the user is visually selecting the first object and not the second object. Finger input and/or other user input may be used in combination with input such as point-of-gaze information in determining which actions are to be taken in system8.

An illustrative system with gaze tracking is shown inFIG. 4. In the example ofFIG. 4, device24is a head-mounted device having a head-mounted support structure116(sometimes referred to as a housing) that is configured to be worn on the head of a user. Device24may include components such as component111. Component111may be, for example, a display. The display and other devices may be mounted in structure116to display computer-generated content in eye boxes120for a user. Rear facing gaze tracking system112may monitor user's eyes104in eye boxes120to determine the direction106of the user's gaze. Additional sensors (e.g. depth sensor114, which may sometimes be referred to as a three-dimensional image sensor) may be used in determining the location and/or other attributes of objects in the user's field of view such as object110ofFIG. 4. Using direction106and/or other information from gaze tracker112and/or other sensors (e.g., a depth sensor and/or other sensors that determine the distance of the user from device24), device24may determine the location of the user's point-of-gaze108on object110.

Object110may be a real-world object (e.g., a body part of the user or other person, an inanimate object with circuitry such as one or more devices24, a non-electronic inanimate object such as a pencil, ball, bottle, cup, table, wall, etc.) or may be a computer-generated (virtual) object that is being presented to the user's eyes104by a display in device24(e.g., a see-through display system or a display system in which virtual content is overlaid on real-world images on the display that have been captured with camera114). Using information on the direction106of the user's gaze and information on the relative position between the user and object110(e.g., information from a depth sensor in device24and/or information on virtual objects being presented to the user), device24may determine when the user's point-of-gaze108coincides with object110.

Arrangements of the type shown inFIG. 4allow a user to interact with real-world content and computer-generated (virtual) content. For example, a user may select an object of interest by directing point-of-gaze108towards that object (e.g., for more than a predetermined dwell time and/or until associated user input such as finger input is received to confirm selection). Using finger device(s)10and/or other equipment in system8, the user may perform operations on the selected object. During use of device24ofFIG. 4and/or at other times, one or more devices in system8(e.g., device(s)10) may be used to gather real-world physical attributes of real-world objects. This sampled real-world information can then be presented to a user with device24and/or other equipment in system8in a computer-generated content environment.

A user may touch items in the user's surroundings while wearing finger devices10. Measurements made with sensors in devices10as the user touches the surfaces of these items can be used in determining the contours of the items. This information can then be combined with optional additional sensor data such as depth sensor data, camera images, temperature data, information on the responses of objects to different amounts of applied force, surface texture data captured with one or more sensors in device10, weight measurements, etc. to determine the physical attributes of real-world items such as size, shape, texture, location, temperature, color and other visual appearance, etc. Examples of sensors that may be used in devices10to measure the contours of items include inertial measurement units, which can track the orientation, position, and/or movement of devices10in three dimensions and force and/or touch sensors in devices10that can sense when a user has contacted the surface of an item. Depth sensors in devices10and/or24may also be used in gathering three-dimensional surface maps (surface contour information) for objects in the user's surroundings. If desired, input from multiple sensors (e.g., a depth sensor in a head-mounted device and a touch sensor in a finger device may be combined to enhance measurement accuracy). For example, a depth sensor may measure the shape of the front face of an object that is facing a user while finger devices may be used in measuring the shape of the opposing rear face of the object.

In general, any suitable sensors may be used in device10to gather information on real-world object physical attributes. These sensors may include, for example, digital image sensors (e.g., cameras operating at visible wavelengths, infrared wavelengths, and/or ultraviolet wavelengths), strain sensors, ultrasonic sensors, direct contact sensors (e.g., capacitive touch sensors, resistive force sensors, capacitive force sensors, and/or other sensors that detect applied force, optical contact sensors, and/or other sensors that detect contact between device10and external surfaces), thermal sensors (e.g., thermocouples, solid state temperature sensors, thermal imaging sensors, and/or other sensors that are configured to measure temperature), three-dimensional sensors (e.g., depth sensors such as structured light depth sensors that emit a set of infrared light beams and that use an infrared image sensor to measure the locations of corresponding dots projected onto nearby three-dimensional objects, binocular vision three-dimensional sensors, etc.), lidar sensors, inertial measurement unit sensors (e.g., accelerometers, compasses, and/or gyroscopes), capacitive sensors that serve as proximity sensors, force sensors, and/or touch sensors, and/or other sensors.

FIG. 5is a perspective view of an illustrative computer-generated environment containing real-world and virtual content. In the example ofFIG. 5, a user is interacting with real-world object130(e.g., a bottle) using finger device10. In particular, the user is using sensors on device10such as touch and/or force sensors or other sensors to measure the surface of object130. By moving the location of finger device10around object130in directions132and134while gathering sensor data from touch and/or force sensors and/or other sensors18, finger device10can map out the location of the surface of object130in three dimensions, thereby determining the global shape and size of the exterior of object130. Local variations in the surface of object130(e.g., texture, recesses, protrusions, etc.) may be sensed during these operations. For example, device10may determine that object130is smooth in location136and textured in location138(as an example).

After gathering information about the shape of object130and other real-world physical attributes, sampled attributes can be played back to the user using the output resources of system8. As an example, a texture that has been sampled with a force sensor or other texture sensitive sensor in device10may be presented to a user using haptic output devices in device10. The played back texture may be presented in a particular portion of the surface of a real-world object. For example, a sampled rough texture or pattern of recesses and/or grooves and/or other virtual haptic content may be presented in area140of smooth area136of object130(e.g., to create a texture associated with a virtual embossed label on a bottle). In addition to providing haptic output associated with virtual objects, system8may provide visual output, audio output, etc. For example, device24may overlay a computer-generated image in area140(e.g., an image that was captured during real-world-object attribute sensing operations on a real-world object with device10or other virtual content). Device24may overlay images on real-world objects by displaying these objects in a head-mounted device, by projecting content onto objects using a display projector, and/or by otherwise overlaying computer-generated images.

Sounds may also be presented in association with computer-generated objects. For example, device10may capture information on the sound of running finger40across a rough fabric. The sampled sound can then be played back to the user with speakers in device24as the user's finger moves across area140. If desired, haptic output may also be presented to a user with device(s)10while the user's fingers are located in the air and are not directly contacting real-world-objects.

If desired, a temperature sensor in device10may be used to measure the temperature of a real-world object. During operation of system8, sampled temperatures may be recreated for the user using a thermoelectric device or other device for providing a desired temperature output (e.g., resistive heating elements, etc.). Sampled real-world object physical attributes such as temperature may be gathered for each location on a real-world object that is touched by finger40. Interpolation techniques may be used to fill in missing temperature samples and/or other real-world attribute measurements.

Sampled real-world-object physical attributes may be stored in a local or online library for later retrieval and use by the user or others. For example, a user may obtain information on a texture and the contour of an object from a shared online library. This information may have been collected and placed in the library by someone other than the user. During operation of the user's system8, the user's system8may use information on the contour of the object and/or other real-world object physical attributes of the object in presenting virtual content to the user (e.g., in presenting virtual content including computer-generated haptic output, visual output, temperature output, audio output, etc.).

As an example, knowing the locations of the surfaces of a real-world cube that were sampled and uploaded to an online library by someone other than the user, the user's system8can overlay virtual visual content on one or more virtual cube surfaces and can provide corresponding virtual haptic output to the user's fingers40using haptic output devices in finger devices10to simulate a sampled texture for those surfaces whenever system8determines that the user's fingers are touching the virtual cube.

If desired, different surfaces of the cube or other object can be provided with different virtual textures using the haptic output devices. As an example, circuitry in device10can supply a first drive signal with a relatively high amplitude and frequency whenever the user's fingers are touching a first side of the cube. Whenever the user's fingers are determined to be touching a second side of the cube, a different texture can be supplied to the user's fingers with the haptic output components in finger devices10. As an example, if the user touches the second side of the cube, a lower-frequency and lower-magnitude drive signal can be used to control the haptic output components in finger devices10. As a result, the user will feel a first texture when touching the first side of the cube and a second texture when touching a second side of the cube. If desired, cubes and other objects can be provided with haptic effects along their edges to create sharp and/or rounded edges, can be provided with haptic effects associated with compliant structures, and/or can be provided with detents, force-feedback simulating motion resistance, clicks simulating depression and/or release of a button with a physical click sensation, and/or other haptic effects. Corresponding visual effects can also be provided using a display in a head-mounted device.

In the example ofFIG. 6, a non-contact sensor is being used to sense surface48of a real-world object such as structure50. The non-contact sensor may be, for example, a laser-based sensor that emits laser light and that uses a corresponding photodetector to monitor reflected portions of the laser light, thereby measuring surface48. Other sensors18(e.g., a three-dimensional structured light sensor or other three-dimensional sensor, a two-dimensional image sensor, a radio-frequency sensor, an acoustic sensor such as an ultrasonic sensor, and/or other non-contact sensor) may also be used in gathering measurements of the physical attributes of surface48(e.g., color, texture, contour shape, etc.).

In the example ofFIG. 7, device10is being used to gather information on structure50using an indirect contact arrangement. In this type of arrangement, the housing walls and other structures of device10do not directly contact surface48, but rather gather information on surface48by virtue of the contact between finger40and surface48. If, as an example, a user moves finger40across a textured portion of surface48, an inertial measurement unit (e.g., an accelerometer, etc.) or other sensor18in device10can sense corresponding vibrations in finger10that reveal the texture. In this way, device10can gather information on the shape of surface48and its response to pressure from finger40, even if device10does not come into direct contact with surface48during physical attribute measurement operations.

Another illustrative arrangement for sampling physical attributes associated with structure50is shown inFIG. 8. In the example ofFIG. 8, device10(e.g., the housing of device10) has a protruding portion such as protrusion10P. Protrusion10P extends past the outermost tip of finger40, which allows device10to use protrusion10P to directly contact surface48and thereby sample physical attributes such as surface texture (e.g., using a force sensor based on a strain gauge in protrusion10P or using an inertial measurement unit (e.g., an accelerometer, etc.). The sensor may measure the texture by measuring movement of device10relative to structure50as finger40drags protrusion10P and therefore device10across the surface of structure50.

In the example ofFIG. 9, the user has moved device10so that device10temporarily covers the user's finger pad. Device10may be, for example, a U-shaped finger device that is sometimes worn on the upper side of finger40. When configured as shown inFIG. 9, device10may directly contact surface48as the user measures physical attributes associated with surface48and structure50. During subsequent haptic playback to recreate the sampled texture, device10may be worn in the finger-pad-covering configuration ofFIG. 9or the configuration ofFIG. 6 or 7in which the user's finger pad is exposed.

Sampled real-world objects may be inanimate objects without mechanical mechanisms or circuitry (e.g., an object such as a bottle, etc.) or may be a button with a movable button member or other device with a movable mechanism and/or circuitry (e.g., a keyboard, a mouse button, etc.). In the example ofFIG. 10, the real-world object being sampled is button131(e.g., an alphanumeric key or other button in an electronic device). During operation of button131, user input such as button press input may be gathered from finger40of a user. When it is desired to sample the physical attributes of button131, device10may be worn on user finger40while user finger40moves over the surface of button131(to measure the surface contours of button131) and while finger40depresses button131.

As shown inFIG. 10, button131may have a dome switch such as switch144mounted on printed circuit142. Guide structures146may help guide movable button member148along the vertical (Z axis) dimension as the user presses and releases moveable button member148. When movable button member148is pressed downward, button member148compresses dome switch144against printed circuit142. When button member148is released, dome switch144pushes button member148upward. By monitoring the state of switch144in button131, the electronic device in which button131is operating may detect the state of button131(e.g., open or closed). At the same time, by using the sensors in device10, the force-versus-distance behavior (force-versus-displacement characteristic) of button131and other information on button131(e.g., size, shape, color, etc.) may be gathered, allowing system8to replicate the performance of button131in a computer-generated environment.

FIG. 11is a graph showing how a user may use device10to gather physical attribute information on button131ofFIG. 10such as information on the shape of button131. A user may, for example, move finger40across button131in direction X while lightly touching the surface of button131. An inertial measurement unit or other position sensor in device10may measure the position of the user's finger in vertical dimension Z and horizontal dimension X as the user moves finger40to different positions across button131. By gathering this position information (see, e.g., curve150ofFIG. 11), device10may determine the shape of the surface of button131.

FIG. 12is a graph showing how device10may gather physical attribute information such as information on the response of button131to various levels of applied force. A user may, for example, press against button member148in a downward (−Z) direction while a position sensor measures the position of device10along dimension Z and a force sensor measures the corresponding amount of force F being applied. This allows device10to gather information on the force-versus-button-depression-distance behavior of button131(see, e.g., curve152ofFIG. 12). The sampled physical behavior of button131can then be replayed to a user in a computer-generated environment by using haptic devices in device(s)10to recreate the sampled behavior of applied force on device10and finger40as a function of displacement.

Sampling may be performed by squeezing an object or by otherwise using one or more fingers such as finger40to apply pressure to the surface of a physical object (e.g., as a user picks up an object, pushes in a particular direction against the object with finger40, etc.). The amounts of pressure (force) applied and the locations and directions of the applied pressures (forces) may be gathered by device10during sampling. These measurements may then be analyzed to determine surface shape, surface rigidity (e.g., response under pressure including response force amount and response force direction) and other attributes. If a movable electronic component such as button131is present, the response of the button to various levels of applied force may be gathered as described in connection withFIGS. 10, 11, and 12. If no button is present, information may be gathered on the location and pressure response of the object's surface. Sampled attributes may then be played back for a user during use of system8(e.g., directional haptic waveforms may be applied to haptic output devices to recreate a force in a desired direction or other directional haptic output, etc.).

If desired, a microphone in device10may gather acoustic measurements (e.g., button click sounds) when button131is being used and a speaker in device24or other equipment in system8may replay the captured sounds for the user (e.g., computer-generated sampled sounds can be used in a computer-generated environment to recreate the sonic experience of interacting with a real-world button). By sampling buttons and other equipment associated with an office environment (e.g., computer keyboards, etc.), system8may recreate a virtual version of a user's office or other equipment with mechanical movable structures. If desired, the behavior of circuits and other components that include other physical attributes (e.g., devices that exhibit particular optical properties, acoustic properties, thermal properties, odors, and/or mechanical properties, etc.) may be sampled and presented to a user in a computer-generated environment.

If desired, textures, surface shapes, visual appearances, temperatures, acoustic properties, and other real-world-object attributes may be edited. For example, one or more individuals may use one or more different pieces of electronic equipment (see, e.g., device10, device24, etc.) to gather measurements of real-world-object physical attributes and these sampled physical attributes may then be cut-and-pasted and/or otherwise edited to create a desired environment.

FIG. 13is a graph of a measured physical attribute MA as a function of distance X (e.g., distance along the surface of a real-world object being measured). In the example ofFIG. 13, a user has moved finger40and device10laterally across the surface of a real-world object. Physical attribute MA has been measured by device10as a function of distance X. As shown inFIG. 13, curve156, which corresponds to measured physical attribute MA includes portion154, which is of interest for subsequent use in a computer-generated environment. Physical attribute MA may correspond to physical surface position (surface contour), temperature, an optical property such as reflectivity, color, etc., local position (texture), rigidity (e.g., deformation amount under pressure), resistance force (e.g., the amount and/or direction of resistance to one or more different amounts of force applied in a particular direction), and/or other physical attribute of a real-world object that is being measured.

FIG. 14is a graph showing the real-world attribute RA of a real-world object. The real-world object associated with attribute RA ofFIG. 14may be an object other than the real-world object that is associated with the graph ofFIG. 13. In this example, illustrative measured attribute MA ofFIG. 13has been measured on a first real-world object, whereas illustrative real-world attribute RA ofFIG. 14is associated with a second real-world object that is different than the first real-world object.

FIG. 15shows how a computer-generated environment may include a sampled portion of the first object that is being played back to the user with the haptic devices of finger device10and/or the display and acoustic devices of device24and/or other equipment in system8. As shown inFIG. 15, virtual output associated with sampled portion154of the first real-world object may be overlaid over the second real-world object associated with curve158. The portion of the second real-world object that has been overlaid by the sample of portion154of the first real-world object may be obscured due to the sample of portion154. For example, the visual, haptic, acoustic, and other attributes of the first real-world object in portion154may obscure underlying visual, haptic, acoustic, and other attributes in portion160of the second real-world object. The portion of the second real-world object that is not overlapped by the recreated first object (see, e.g., curve158ofFIG. 15in the region other than portion154) may be directly sensed by the user. For example, the first object may be a piece of fabric with a texture and color that is of interest to a user. This texture and color may be sampled using device10and overlaid on a given portion of a second object such as a bottle. When the user interacts with the bottle, the texture and color of the fabric may replace the bottle's normal characteristics in the given portion of the bottle, whereas other portions of the bottle may have the bottle's original texture and color.

Sampled physical attribute cutting-and-pasting operations of the type described in connection withFIGS. 13, 14, and 15may be performed by a user and/or by others. Sampled content that is to be incorporated into a computer-generated environment may be selected from a shared online library and/or from a user's personal library. Sensor information such as information that a user gathers with device10may be shared with others using the online library subject to the user's permission and/or other safeguards. If desired, cut-and-pasted attributes may include response force (e.g., information on the amount of resistance experienced by finger40in response to applying a given amount of force in a given direction at a given location). Haptic output can be used to recreate response force feedback that simulates user interaction with a sampled real-world object. In this way, real-world resistance forces (responses to applied forces) may be cut-and-pasted into virtual environments. In scenarios in which temperature measurements are sampled, cut-and-pasted temperature readings can be merged into virtual environments. In scenarios in which finger bending forces are measured during user interactions with real-world objects (e.g., using a finger bending sensor in portion10B of device10ofFIG. 2), finger bending characteristics can be cut-and-pasted into a virtual environment. During use of system8, the user may experience recreated temperatures, finger bending forces, and other physical attributes based on the sampled physical attributes.

FIG. 16is a flow chart of illustrative operations that may be associated with using system8.

During the operations of block160, the real-world-object physical attributes of physical objects may be measured using sensors18in device10and/or other sensors in system8as a user and/or others interact with real-world objects. The real-world objects may include inanimate objects without moving parts, buttons and other objects that have mechanical mechanisms that move in response to finger pressure and/or other applied forces, electronic circuitry (e.g., a touch sensitive device), and/or other real-world objects.

Sensors18and/or other sensors in system8may measure surface contours (e.g., some or all of the overall (global) shape of the surface of an object), may measure local surface attributes (e.g., texture, localized protrusions and/or grooves, etc.), may measure optical characteristics (e.g., color, visual pattern, reflectivity, absorption, and transmission at visible, infrared, and/or ultraviolet light wavelengths), electrical properties (radio-transparency, frequency resonances, surface and interior structures associated with the absorption, reflection, and/or transmission of electromagnetic signals at non-light wavelengths), acoustic attributes, resilience (e.g., stiffness, flexibility, elasticity, hardness, and/or other material attributes), weight, torque under various usage conditions (e.g., torque on a user's fingers when a wand or baseball bat is being swung back and forth), friction (e.g., a coefficient of static friction or a coefficient of dynamic friction as measured by a shear force sensor in device10), force-versus-displacement (force-versus-distance) behavior (e.g., change in surface location and/or other attributes as a function of applied finger pressure or other applied force), etc.

Information that is gathered during the operations of block160may, if permitted by a user, be shared by uploading this information to an online database. Configurations in which sampled information is stored locally or is otherwise not shared with others may also be used. If desired, different electronic devices and/or different types of electronic devices may be used in gathering the information during the operations of block160than are used by users in playing back this information during use of system8. Arrangements in which the same type of device and/or the same electronic device is used in both sampling and playing back information may also be used.

The real-world objects that are sampled during the operations of block160may include household objects (cups, bottles, furniture, clothes, and other items), may include office equipment (computers, keyboards, accessories such as computer mice, etc.), may include video game equipment (e.g., prop swords or wands), may include sports equipment (e.g., rackets, balls, pucks, clubs, bats, sticks, and/or other sports equipment), and/or other real-world objects.

During the operations of block162, system8(e.g., device10and device(s)24) may provide output to create a computer-generated environment. The output may include output corresponding to the real-world-object physical attributes that were measured during the operations of block160(e.g., sampled information may be played back for a user). For example, the computer-generated environment may include surface shapes, textures, object colors and other visual appearance attributes, sounds, force-versus-displacement characteristics and other moving mechanisms characteristics, weights, temperatures, and/or other real-world-object physical attributes that were measured using device10, device(s)24, and/or other electronic equipment with sensors and that are being presented to the user using device(s)10and/or device(s)24. Measured attributes may be retrieved from local storage and/or from cloud storage (e.g., an online library that is accessed through a communications network with local and/or remote links formed using wireless and/or wired communications paths). Recreated physical object attributes can be overlaid on real-world objects. For example, a sampled texture may be recreated on a portion of a bottle or other real-world object, thereby replacing the real-world attributes of the object with the sampled attribute(s). If desired, recreated physical object attributes can be recreated in free space (e.g., as a user's fingers are moving through the air). Visual content may be overlaid on real-world objects by displaying computer-generated images in a head-mounted device or other device that displays computer-generated content on top of real-world images and/or by projecting visual content onto real-world objects using a projector (e.g., a projector in device10and/or a projector in device24). A user may provide user input using finger devices10and/or other devices24. For example, a user may provide user input such as a swipe gesture input using a finger device and this input may be used to move a virtual object displayed in computer-generated content that is being viewed by the user with a display in a head-mounted device. The haptic output and other output in the computer-generated environment that is presented based on sampled real-world-object physical attributes may be presented to the user as the user is using a finger device or other device to provide user input. For example, haptic output or other output based on sampled real-world-object weight characteristics may be provided as a user is moving a virtual object with a finger device.

Physical Environment

Examples of CGR include virtual reality and mixed reality.

Virtual Reality

Mixed Reality

Augmented Reality

Hardware

As described above, one aspect of the present technology is the gathering and use of information such as sensor information. The present disclosure contemplates that in some instances, data may be gathered that includes personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data can include demographic data, location-based data, telephone numbers, email addresses, twitter ID's, home addresses, data or records relating to a user's health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, username, password, biometric information, or any other identifying or personal information.

Therefore, although the present disclosure broadly covers use of information that may include personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing personal information data. That is, the various embodiments of the present technology are not rendered inoperable due to the lack of all or a portion of such personal information data.