Device for physical interaction between remotely located users

An electronic device for touch translation includes a body and pins extending therefrom and including couplings to facilitate movement of a first portion relative to a second portion. The pins are controllable to move the first portion relative to the second portion and to control force applied by the pins on an external object. Heads are disposed on the pins, which heads are greater in width than the pins and are movable relative to the pins about respective couplings. Sensors cooperating with the pins detect forces applied to the pins and a communication subsystem communicates over a network, with a remote electronic device. A controller, based on detected forces, transmits signals to the remote electronic device to control the remote electronic device, and actuates pins to control the relative movement of the portions based on signals received from the remote electronic device.

FIELD OF TECHNOLOGY

The present disclosure relates to devices for interaction between, for example, people located remotely from each other.

BACKGROUND

Electronic devices, such as smart phones, tablet computers, laptop computers, and desktop computers have gained widespread use for a variety of functions including communications functions. Video communication functions, utilizing video chat applications, are commonly used both for business and for personal use between people located remotely from each other, for example, between parents and children living in different locations, between spouses when one or both are travelling, between colleagues working in different locations, and so forth. Thus, interactions between people are commonly carried out remotely.

Interactions between people utilizing electronic devices for communication functions are limited, for example, to voice, video, or both voice and video communication.

Head-mounted displays may also be utilized for virtual interaction between individuals to provide a more realistic interaction. Such interactions, however, are only virtual and are limited to interaction in a virtual space.

Improvements in electronic devices to provide further interaction capabilities between people located remotely from each other are desirable.

SUMMARY

An electronic device for touch translation is provided. The electronic device includes a body, a plurality of pins extending from the body, the pins including couplings to facilitate movement of a first portion of the pins relative to a second portion of the pins, the pins being controllable to control the movement of the first portion relative to the second portion and to control a force applied by the pins on an external object, sensors cooperating with the pins to detect forces externally applied to the pins, a communication subsystem for communication, over a network, with a remote electronic device, and a controller coupled to the pins, the sensors, and the communication subsystem. The controller controls the electronic device to, based on detected forces externally applied to the pins, transmit a signal to the remote electronic device for the control of the remote electronic device, and to, based on signals received from the remote electronic device, actuate ones of the pins to control movement of the first portion relative to the second portion and to control the force applied by the pins on the external object.

DETAILED DESCRIPTION

For simplicity and clarity of illustration, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. Numerous details are set forth to provide an understanding of the examples described herein. The examples may be practiced without these details. In other instances, well-known methods, procedures, and components are not described in detail to avoid obscuring the examples described. The description is not to be considered as limited to the scope of the examples described herein.

The following describes an electronic device and a method for touch translation. The electronic device includes a body, a plurality of pins extending from the body, the pins including connections or couplings to facilitate movement of a first portion of the pins relative to a second portion of the pins, the pins being controllable to control the movement of the first portion relative to the second portion and to control a force applied by the pins on an external object, sensors cooperating with the pins to detect forces externally applied to the pins, a communication subsystem for communication, over a network, with a remote electronic device, and a controller coupled to the pins, the sensors, and the communication subsystem. The controller controls the electronic device to, based on detected forces externally applied to the pins, transmit a signal to the remote electronic device for the control of the remote electronic device, and to, based on signals received from the remote electronic device, actuate ones of the pins to control the movement of the first portion relative to the second portion and to control the force applied by the pins on the external object.

A simplified block diagram of an example of an electronic device100for touch translation is shown inFIG. 1. The electronic device100includes multiple components, such as a main processor102that controls the overall operation of the electronic device100. The electronic device100may be mounted to another object or device, may include mounting brackets or geometry to facilitate mounting to another object or device, or may be in the form of a sheet for resting on a surface. According to one example, the electronic device100is incorporated into a case for another electronic device such as a smartphone or tablet computer. Alternatively, the electronic device100may be incorporated or integrated into another electronic device such as a portable electronic device, smartphone, or tablet computer. The electronic device100may also be generally transparent and may be overlaid on a display of another electronic device such as a smartphone or tablet computer. Thus, the display on which the electronic device100is overlaid may be a touch-sensitive display. Optionally, the electronic device100is manually removable when not in use. Alternatively, the electronic device100may be opaque or partially opaque and is included in a cover that covers part or all of the display118when utilized. For example, the electronic device100may be incorporated into a phone case that is manually located over the display to close the case over the display prior to touching the electronic device100, which may include holding the electronic device100to the face of the user.

The main processor102interacts with other components of the electronic device100, including, for example, a temporary storage device104, a memory106, an auxiliary input/output (I/O) subsystem108, a communication subsystem110, a power source112, and, optionally, other subsystems114. Additionally, the main processor102interacts with a controller116that is coupled to actuators118that are utilized to control movement of pins, also referred to as fingers, about connections or couplings within the pins.

The force sensors120are associated with the pins, for example, are located at the ends of the pins of the electronic device100to detect external forces that are applied to the pins, such as forces from a user's hand, finger, thumb, face, appendage, or other items held by a user applying force to the pins of the electronic device100. The force sensors120may be disposed in the pins, on the pins, under the pins, or any suitable combination of in, on, and under the pins to detect forces on the pins. Thus, an external force applied to the pins of the electronic device100is detected utilizing the force sensors120. The actuators118may also be utilized to apply a force, by the pins, on an external object, such as the user's hand, finger, thumb, face, appendage, or other items, held by a user applying force to the pins.

The temporary storage device104may be, for example, Random Access Memory (RAM) that stores data that is processed by the main processor102. The memory106, such as flash memory, is utilized for persistent storage. The memory106may be utilized to store an operating system and software programs or components that are executed by the processor102.

The optional auxiliary input/output (I/O) subsystem108may include an interface through which, for example, a USB controller or other peripheral device may be connected. Other input/output subsystems may also be utilized as well as other communications.

The communication subsystem110receives signals from a communication device such as a portable electronic device, smart phone, tablet computer, laptop or other device (not shown) and sends signals through the communication device to which the electronic device100is coupled. Thus, for example, the signals from the force sensors120or other commands from the main processor102may be sent via the communication subsystem110. The communication subsystem110is also responsible for receiving signals via the communication device for processing by the main processor102to cause actuation of the actuators118, via the controller116, in response to signals from the communication device.

The power source112may be one or more of rechargeable batteries, capacitors, inductive charging, inductive power, fuel cells, a port to an external power supply to power the electronic device100.

The systems and subsystems that interact with the main processor102and are described herein are provided as examples only. Other subsystems114may also interact with the main processor102.

Although not shown in the block diagram ofFIG. 1, the electronic device100may optionally include other devices and subsystems. For example, the electronic device100may include a display device or display devices for displaying information such as pictures or other information on the pins, one or more speakers for audio output, one or more cameras for capturing images, which may include video, short-range communications, proximity sensors, and other suitable devices or subsystems.

Referring toFIG. 2, a system for touch translation including the electronic device100is shown. In this example, the electronic device100communicates with a similar electronic device200that is located remotely from the electronic device100by sending signals to the remotely located electronic device200via the communication device202to which the electronic device100is coupled, through a network250and through a remotely located communication device204. The electronic device100also receives signals from the remotely located electronic device200via the communication device202, the network250, and through the remotely located communication device204.

As indicated above, the communication device202may be a portable electronic device, smart phone, tablet computer, laptop or other device that is in communication with the electronic device100via the communication subsystem110of the electronic device100. The electronic device100may optionally be physically coupled to the communication device202. For example, the electronic device100may be coupled to a back side of the communication device202.

Similarly, the remote communication device204may be a portable electronic device, smart phone, tablet computer, laptop or other device that is in communication with the remote electronic device200via a communication subsystem of the remote electronic device200. The remote electronic device200may optionally be physically coupled to the communication device204.

The network250may include the internet and may include a cellular network in addition to the internet or as an alternative to the internet. Several communication devices may communicate through the network250. Other communications may also be utilized, including for example, near field, Bluetooth®, WiFi, optical, radio, or a combination of communications.

Thus, the electronic device100is operable to communicate with the remote electronic device200. When a communication session begins, signals are transmitted from the electronic device100to the remote electronic device200in response to detecting an externally applied force on the pins of the electronic device100. The signals are sent to the remote electronic device200to control the remote electronic device200. In response to receipt of signals at the electronic device100, from the remote electronic device200, the actuators are controlled to control movement of the pins of the electronic device100and force applied by the pins of the electronic device100on an external object, such as a user's hand, finger, thumb, face, appendage, or other items, held by a user applying force to the pins. Thus, a force applied by a user on the electronic device100is determined and, movement of the pins of the remote electronic device200is controlled and a resulting force is applied by the remote electronic device200. Similarly, a force applied by a remote user on the remote electronic device200is determined and a resulting force is applied by the electronic device.

A user pressing on the pins on the electronic device100at the same time that a remote user presses on the remote electronic device200, feels the return force caused by the user pressing on the remote electronic device200.

A perspective view of one example of an electronic device is shown inFIG. 3AthroughFIG. 3C. The electronic device100includes a body302in which the components illustrated inFIG. 1are disposed. The body302may be rigid. Alternatively, the body302may be flexible while still providing protection for the components shown inFIG. 1. The plurality of pins304extend from the body302. In the present example, the pins304extend generally linearly away from the body302. Each pin304includes coupling306such that a distal portion308or outer portion of the pin304is moveable relative the proximal portion310or inner portion of the pin. The proximal portion310is coupled to the body302while the distal portion308is coupled at the coupling306to the proximal portion310.

For the purpose of the present example, the distal portion308is moveable relative to the proximal portion310, toward and away from the body302. Movement of a pin304and any force applied by a pin on an external object that is in contact with the end of the distal portion308of the pin304or a cover or membrane coupled to the distal portion308of the pin304, is controlled by an actuator, such as a linear actuator, which may be, for example, a hydraulic actuator or pneumatic actuator. The linear actuator is coupled to the controller to thereby control sliding movement of the distal portion308relative to the proximal portion310of the pin304, and force applied by the pin304. In this example, coupling comprises a telescoping coupling and the sliding movement of the distal portion308relative to the proximal portion310is a telescoping movement.

For the purpose of this example, the pins304are covered by a flexible, elastic membrane312such as a latex, flexible PVC, CyberSkin® or a combination of flexible, elastic materials. Thus, in this example, the pins304and membrane312are part of a user interface of the electronic device100.

The flexible, elastic membrane312may also be comprised of multiple layers of materials. For example, the flexible, elastic membrane312may include a first layer314that couples to at least some of the pins304, for example, by a mechanical interlock with sockets in the layer or an adhesive layer that facilitates application of forces by the pins312, away from the body302and toward the body302. The flexible, elastic membrane312may include a second layer316of, for example CyberSkin®, and a third layer318of, for example, a very thin latex. The very thin latex may be replaceable.

Alternatively, heads320may be disposed on the ends of the pins304and the heads are larger in diameter than the body of the pins304, as illustrated in the example ofFIG. 3DandFIG. 3E. In this example, no membrane312is present. The heads320may be a different material or materials than the pins304.

Referring again toFIG. 3AthroughFIG. 3C, the plurality of the pins304extend from the body302, in a dense array of pins304that are each, individually actuatable. In addition to being actuatable, the pins304are depressible by an externally applied force. Such an externally applied force is detected utilizing the force sensors120. The force sensors may be coupled to the pins such that each pin is associated with a respective force sensor for detecting the externally applied force on that associated pin.

In accordance with the present example, the pins304are small relative to a human finger, thumb, hand, appendage, or face and are disposed in a dense array on the body302such that an external force exerted, for example, by a human finger, is exerted on a plurality of the pins304. Thus, force may be applied to tens or hundreds of pins304by a user's finger pressing on the electronic device100. As a result of the relatively high number and density of pins304, a force is applied on the pins304, which together are moved in the shape of the finger or other object that applies the force.

Referring again toFIG. 2, when an external force is applied to the pins304, sufficient to cause the distal portion308of some of the pins304to move toward the body302, signals are transmitted to the remote electronic device200that is in communication with the electronic device100. When no external force is applied to pins at the remote electronic device200, the pins at the remote electronic device200that correspond to the pins304to which the force is applied at the electronic device100, are moved. The corresponding pins at the remote electronic device200are moved by moving the distal portion of the pins away from the body.

Thus, the ends of the pins304are moved toward the body302at the electronic device100to provide a depression in the surface that generally follows the contour and surface profile of the object, such as a finger, that applied the force to the pins304. At the remote electronic device200, ends of the corresponding pins are moved away from the body to form a projection that generally follows the contour and surface profile of the object that applied the force to the pins304. The projection formed at the remote electronic device200is formed by the pins covered by the elastic membrane, giving the general appearance of the object that applied the force to the pins304at the electronic device100.

In this example, the distal ends308of the pins304are moved at the electronic device100to form a depression in the surface of the membrane312, and the distal ends of the pins are moved at the remote electronic device200to form a corresponding projection in the surface of the membrane. Thus, the pins at the remote electronic device200are moved in the opposite direction as the pins at the electronic device100to generally form an inverse profile.

When an external force is applied to the pins304at the electronic device100, and an external force is applied to corresponding pins at the remote electronic device200, the corresponding pins at the remote electronic device200apply a force to the external object applying a force at the remote electronic device200. Similarly, the pins304at the electronic device100apply a force on the object applying the external force at the electronic device100. The force applied by the pins at the remote electronic device200to the external object, generally corresponds in magnitude to the external force applied to the pins304at the electronic device100. The force applied by the pins304to the external object at the electronic device100generally corresponds in magnitude to the external force applied to the pins at the remote electronic device200.

A flowchart illustrating a method of controlling an electronic device, such as the electronic device100is shown inFIG. 4. The method may be carried out by software executed, for example, by the main processor102of the electronic device100. Coding of software for carrying out such a method is within the scope of a person of ordinary skill in the art given the present description. The method may contain additional or fewer processes than shown or described, and may be performed in a different order. Computer-readable code executable by at least one processor to perform the method may be stored in a computer-readable medium, such as a non-transitory computer-readable medium.

A communication session is initiated402between the electronic device100and the remote electronic device200. The communication session is started by one or both of the electronic device100and the remote electronic device200. To initiate the communication session, the electronic device100and the remote electronic device perform a handshake process, for example, in response to user selection of an option to begin communicating with the remote electronic device200. The communication session is secured utilizing known secure communications techniques, including, for example, encryption and decryption to provide security in transmission.

Initiation of a communication session may also include user authentication or identification. For example, knowledge-based identification, such as a passcode or a personal identification number, may be utilized. Alternatively or in addition, biometric identification, such as fingerprint, facial recognition, palm print, or geometry, or other biometric identification may be utilized. Such biometric identification may be carried out by pressing a hand, face, or other body part against the interface of the electronic device100. A comparison is then made with stored data relating to the user's identification to confirm that the user is an authorized user of the electronic device100. The electronic device may also identify contours of details of the hand or other body part and, optionally measure temperature to confirm the temperature of the person for use in authentication. Such biometric identification utilizing an imprint of a hand or other body part against the interface increases security over biometric identification methods of other devices. The length of time that the communication session lasts may be limited. For example, after a threshold period of time, the communication session may be discontinued unless authentication is repeated.

During the communication session, externally applied forces on the interface of the local electronic device100are detected utilizing the force sensors120. In response to detecting an externally applied force on the interface at404, signals are transmitted to the remote electronic device200at406to actuate the actuators to control movement of portions of the pins about the connections or couplings to thereby control movement and forces applied by the pins of the remote electronic device200.

Signals are also received at the local electronic device100. The signals are received from the remote electronic device200in response to externally applied forces that are detected at the remote electronic device200.

In response to receipt of signals at the local electronic device at408, the actuators118are actuated at410to control movement of portions of the pins about the couplings to thereby control movement and forces applied by the pins304of the local electronic device100.

Because the pins304of the electronic device100include couplings306to facilitate movement of the distal portion308relative to the proximal portion310, the pins304are movable toward and away from the body302and are operable to apply a force to an object touching the pins304. In addition, the pins304are controlled to form a shape, such as a projection, that generally follows the contours and surface profile of an object touching the interface of a remote device that is in communication with the electronic device100. Utilizing the movement of the pins304and force application, the electronic device100, in cooperation with a remote electronic device200, simulates touch between two people that are each utilizing a respective one of the electronic devices.

Utilizing such electronic devices100,200, touch contact is simulated to give the users the perception of touch. For example, a first user that presses on the pins304of the local electronic device100, while a second user presses the pins of the remote electronic device200, which is in communication with the local electronic device100, perceives touch contact with the second user. For example, if both users rest their hands on the respective interface, the first user perceives that he or she is resting a hand on a hand of the second user. Similarly, the second user perceives that he or she is resting the hand on the hand of the first user. In another example, facial contact is perceived when the users press their faces against the interface.

Latency introduced from various sources such as transmission time, processing time, and actuation, may be of the order tens of milliseconds or greater. To reduce the problems introduced by latency and thus a lack of synchronization between the two electronic devices100,200, the software stored, for example, in the memory106at each electronic device, is utilized to smooth out actions and reactions to generally maintain the simulation of touch and facilitate perception of touch by each user.

The communication session or parts thereof may optionally be recorded by storing information relating the communication session. For example, received signals from an electronic device may be recorded by storing related information in the memory, such as the memory106. For example, the signals received from a remote electronic device in response to a user placing his or her hand on the remote electronic device, may be stored and utilized later to reproduce the simulation of touch after the communication session has ended.

In the above-described method, the electronic device100enters a communication session with the remote electronic device200to simulate touch at both the electronic device100and at the remote electronic device200. Optionally, the application of force and movement of the pins may occur at an electronic device, such as the electronic device100, independent of a communication session with another electronic device. Signals may then be transmitted, for example, by a social networking platform, to share the signals with a recipient or with multiple recipients. For example, a touch may be broadcast to multiple recipients. In one example, a user places his or her lips on the electronic device and kisses the interface. The signals resulting from the kiss may be stored remotely for another user or users to obtain. For example, a movie star may make a kiss available for a plurality of fans to receive on their own electronic devices.

In another example, signals may be received at the electronic device100from a plurality of remote electronic devices and the signals may be combined, modified, averaged, or any combination thereof. Referring to the example of the movie star making a kiss available for a plurality of fans, the fans, in return, may make a kiss available for the movie star. Thus, a plurality of touches may be, for example, averaged to provide a combined response. Alternatively, a single, representative response, which may be from a single user, may be provided, where that response falls within a predetermined range of feedback.

According to another example, the electronic device100may be utilized by moving the user interface and applying forces to a user to simulate interaction with a virtual person or object.

A simplified block diagram of another example of an electronic device100for touch translation is shown inFIG. 5. In the example shown and described inFIG. 1, the electronic device100is utilized in conjunction with a communication device, such as a smartphone, tablet computer, or laptop computer, in order to communicate over a network with a remote electronic device. In the example shown inFIG. 5, the electronic device500may be utilized without connecting to a second device. Thus, the electronic device500in this example is operable to communicate over a network without the use another communication device.

Many of the elements or components referred to inFIG. 5are similar to the elements or components inFIG. 1. For simplicity and clarity of illustration, the reference numerals are raised by 400 to indicate corresponding or analogous elements.

The electronic device500includes multiple components, such as a main processor502that controls the overall operation of the electronic device100. As indicated, the electronic device500is operable to communicate, over a network, with a remote electronic device. The electronic device500in this example, may be any suitable size, depending on the application or intended use.

The main processor502interacts with other components of the electronic device500, including, for example, a temporary storage device504, a memory506, an auxiliary input/output (I/O) subsystem508, a communication subsystem510, a power source512, and, optionally, other subsystems514. Additionally, the main processor502interacts with a controller516that is coupled to actuators518that are utilized to control movement of the pins about connections or couplings within the pins.

The functions of many of the components are similar to those described with reference toFIG. 1and are therefore not described in detail again herein.

In the present example, communication functions are performed through the communication subsystem110. Data received by the electronic device100is decompressed and decrypted by a decoder522. The communication subsystem510receives signals from and sends messages to a network (not shown).

The main processor502may also interact with other components such as a speaker524, a microphone526, a display528, one or more cameras530, and short-range communications532.

The speaker524outputs audible information converted from electrical signals, and the microphone526converts audible information into electrical signals for processing. The display528may be any suitable display or displays for displaying information, for example, on the pins. The display528may project an image or may be embedded in the pins or on heads that are disposed on the pins in order to display information, such as images, on the pins.

The camera or cameras530are utilized to obtain images or video of the user of the electronic device500. Optionally, the cameras530may be utilized to obtain images or video of the user's surroundings as well. Each of the cameras includes the functional components for operation of the camera, including the lens, the image sensor, and, optionally, a light sensor and light source, such as infrared light emitting diodes (LEDs). The cameras may be one or more of visual light cameras, 3D sensing cameras, light field cameras, forward looking infrared cameras, near infrared cameras, ultraviolet cameras, or other imaging devices.

The short-range communications532may be utilized to perform various communication functions. For example, the short-range communications532may include Bluetooth or infrared (IR) communications capability for communicating with another electronic device, a peripheral device, or accessory.

Referring toFIG. 6, a system for touch translation including the electronic device500is shown. In this example, the electronic device500communicates with a similar electronic device600that is located remotely from the electronic device500by sending signals to the remotely located electronic device600via the network250. The electronic device500also receives signals from the remotely located electronic device600via the network250. Thus, no communication device is utilized in the present example of the electronic device500for communication via the network250. The electronic device500is operable to communicate directly over the network250.

Thus, the electronic device500is operable to communicate with the remote electronic device600. When a communication session begins, signals are transmitted from the electronic device500to the remote electronic device600in response to detecting an externally applied force on the pins of the electronic device500. The signals are sent to the remote electronic device600to control the actuators and thereby control the movement of pins and forces applied by the pins at the remote electronic device600. In response to receipt of signals at the electronic device500, from the remote electronic device600, the actuators are controlled to control movement of the pins of the electronic device500and forces applied by the pins of the electronic device500on an external object, such as a user's hand, finger, thumb, face, appendage, or other items, held by a user applying force to the pins. Thus, a force applied by a user on the electronic device500is determined, movement of the pins of the remote electronic device600is controlled, and a resulting force is applied by the remote electronic device600. Similarly, a force applied by a remote user on the remote electronic device600is determined and a resulting force is applied by the pins of the electronic device500. The operation of the electronic device500may be similar to the operation of the electronic device100and thus, the operation is not further described herein.

The method described above and shown inFIG. 4is also applicable to the electronic device shown inFIG. 5. The method may be carried out by software executed, for example, by the main processor502of the electronic device500. Details of the method shown inFIG. 4and described above are also applicable to the electronic device500and are therefore not described again herein.

As with the electronic device100, the pins of the electronic device500include couplings to facilitate movement of portions of the pins and to facilitate application of a force to an object touching the pins. In addition, the pins are controlled to form a shape, such as a projection, that generally follows the contours and surface profile of an object touching the interface of the remote device600that is in communication with the electronic device500. Utilizing the movement of the pins and force application, the electronic device500, in cooperation with a remote electronic device600, simulates touch between two people that are each utilizing a respective one of the electronic devices. Utilizing such electronic devices500, touch contact is simulated to give the users the perception of touch.

The force that is applied by the electronic device500on the user or the force that is applied by the remote electronic device600on the remote user is controlled such that only forces that are within a predetermined range are transmitted. For example, signals that result from forces that are deemed to be outside of a safe range, for example, that may result in blunt trauma or sharp forces that may cause injury are not transmitted or are not utilized by the receiving electronic device. Alternatively, such forces may be altered, for example to reduce the speed of the force, reduce the sharpness, reduce the magnitude, or any suitable combination of these alterations in force.

One or both the remote electronic device600and the local electronic device500may compare the force or value representative of the force to a threshold limit to determine whether the force is within predetermined safety limits. This method may be carried out, for example in the method shown and described above with reference toFIG. 4. Thus, the method is carried out by software executed, for example, by the main processor502of the electronic device500. Coding of software for carrying out such a method is within the scope of a person of ordinary skill in the art given the present description. Additional or fewer processes may also be performed and computer-readable code executable by the processor502to perform the method may be stored in memory506.

Thus, as part of the process, for example, at404ofFIG. 4, the processor502of the electronic device500may compare the force or a value representative of the force to a threshold limit stored in memory506. In response to determining that the force or value meets or exceeds the threshold limit, the processor502of the electronic device500does not transmit the associated signals to the remote electronic device600such that the force is not applied to user of the remote electronic device600. On the other hand, in response to determining that the force or value is less than the threshold limit stored in memory506, the signals are transmitted and the force is applied to the user of the remote electronic device600.

In addition, as part of the process, for example, at408ofFIG. 4, the electronic device500may compare signals received from the remote electronic device600to predetermined values prior to actuating actuators to apply forces, by the local electronic device500, to the user. In response to determining that the force or value meets or exceeds the threshold limit, the processor502of the electronic device500does not actuate the actuators such that the force is not applied to user of the local electronic device500. On the other hand, in response to determining that the force or value is less than the threshold limit stored in memory506, the force is applied to the user of the remote electronic device600.

The size or shape, such as the width across which the force is applied may also be utilized such that forces from very sharp objects are not transmitted to the user. For example, the threshold limit may vary depending on the dimensions, such as width, across which the force is applied to the pins of the electronic device500. Thus, for example, the electronic device500may maintain a lookup table in memory506and the threshold limit that is utilized for the comparison is identified from the lookup table and is dependent on dimensions including length and width of the applied force.

Optionally, the output may be scaled relative to the input such that, for example, inputs provided by a baby or a person with neuromuscular damage, which are by nature relatively weak, are amplified by some factor. Such scaling is also useful where one of the users desires touch that is stronger or weaker than the other user normally provides.

A perspective view of another example of an electronic device is shown inFIG. 7. Although the electronic device500described with reference toFIG. 5is referred to in the present description with reference toFIG. 7, the present description is equally applicable to the electronic device100described herein with reference toFIG. 1.

The electronic device500includes a body702in which the components illustrated inFIG. 1are disposed. The body702may be rigid. Alternatively, the body702may be flexible while still providing protection for the components therein. The plurality of pins704extend generally away from the body702. In the present example, the pins704each include a plurality of couplings706, which are articulating joints. The couplings706in each pin704may include more than one type of articulating joint to facilitate various types of movements of portions of the pins704. Although three joints706are illustrated in the Example ofFIG. 7, fewer or more joints may be utilized to facilitate movement of the portions of the pins704. The articulating joints may include, for example, hinge joints, prismatic or sliding joints, revolute joints, or any suitable combination of joints or other couplings. The couplings together provide a linkage to facilitate movement in more than one axis. Thus, the portions of the pins704are coupled together about couplings to facilitate movement in all directions, as shown inFIG. 7B. In addition, all or a subset of the pins704may be coupled to the body702utilizing a coupling, for example, to facilitate gliding movement of the pins704relative to the body702.

Movement of the pins704about the couplings706may be controlled by wires708, for example, that couple portions of the pins704to the body702or to other portions of the pins704, as illustrated inFIG. 7C. The wires708may be controlled by actuation of the actuators518. For example, the wires may be pulled when actuators518of the electronic device500are actuated to move the heads714on the pins704about a coupling. Optionally, some or all of the pins704may be disposed on rollers712on the body714to facilitate movement of the pins704relative to the body714, as shown inFIGS. 7A and 7D.

The pins704are movable along the body702in a sliding or gliding motion, movable, toward and away from the body702. Different portions of the pins704are also moveable relative to the body702to facilitate movement of the pins704in other directions. Movement of the pins704and any force applied by a pin704on an external object that is in contact with a head714on the pin704, is controlled by multiple actuators that cooperate to control movement and force applied by the pin704. Thus, the heads714on the pins704are movable in three dimensions facilitating flexion, extension, rotation, adduction, abduction, or any combination thereof, of the pins704.

In this example, heads714are mounted on the ends of the pins704. The heads714are geometrically shaped to include a plurality of facets716and are coupled to the pins704to facilitate rolling of the heads714relative to the pins704to select which of the facets716is exposed or directed outwardly. Thus, the head714may be rolled to expose any one of, for example, four facets716depending on the application. Each facet716may have different material properties to provide different sensations to the touch, as illustrated inFIG. 7EandFIG. 7F. For example, the four facets716may include one facet covered by a material such as latex or CyberSkin®, a second facet covered with very fine wisps of hair or hair-like material on a silicone or material base, a third facet covered by a more dense coat of hair; and a fourth facet covered by a cloth material. Thus, depending on which of the facets716is exposed at the end of each of the pins704, the heads714are utilized to simulate the feel of different surfaces or textures.

As in the embodiment described herein with reference toFIG. 3, a plurality of the pins704extend from the body702, in a dense array of pins704that are, individually actuatable and each individual actuator within the pins is actuatable. In addition to being actuatable, the pins are depressible or flexed by an externally applied force. Such an externally applied force is detected utilizing the force sensors520. The force sensors520are disposed on the pins704to detect externally applied static or dynamic forces including, for example, compressive force, frictional force, tensile force, torsion, and any combination of such forces.

The pins704are small relative to a human finger, thumb, hand, appendage, or face and are disposed in a dense array on the body702such that an external force exerted, for example, by a human finger, is exerted on a plurality of the pins704. Thus, force may be applied to tens of pins704, hundreds of pins704, or more, by a user's finger pressing on the electronic device500. As a result of the relatively high number and density of pins704, such a force is applied on the pins704, which together generally follow the contour and surface profile of the object that applied the force to the pins304.

Because, the pins704are moveable in multiple axes in response to an externally applied force or in response to signals received via the communication subsystem510, for example, and are operable to apply force in multiple directions against an external object, more complex touch interaction in which forces are applied in more than one direction or plane may be simulated. In addition, further contours and movement of the object that applied the force to the pins304may be formed in a similar, remote electronic device in communication with the local electronic device500.

According to one example, an electronic device500may be held up to the throat of a user while a doctor manipulates a remote device that the doctor is using to simulate the feel of the glands of the user of the electronic device500. Thus, the doctor manipulates the pins on the remote device such that the pins704apply a light force against the glands of the user. The size and contours of the glands may be determined by the doctor based on the reaction forces on the pins704, which are utilized at the remote device to simulate the throat, which is the object to which the force is applied and that is applying the reaction forces against the pins704. With sufficient sensitivity, a doctor can also detect the pulse as the pulse is simulated at the remote device.

Alternatively, the electronic device500may be utilized to simulate a physical handshake between remotely located users, hand holding, cheek touching, and any other suitable touch interaction.

In addition, with movement of the heads714relative to the body702, for example, in a sliding or gliding motion, a rubbing or friction force may be simulated. To facilitate simulation of rubbing or friction, the electronic device500may optionally introduce noise into signals sent to a remote electronic device or received from the remote electronic device such that the movement of the heads714relative to the user is not smooth.

Referring again toFIG. 6, when an external force is applied to the pins704, sufficient to cause flexion or movement of the pins704relative to the body702, signals are transmitted to the remote electronic device600which is in communication with the electronic device500. When no external force is applied to pins at the remote electronic device600, the pins at the remote electronic device600that correspond to the pins704to which the force is applied at the electronic device500, are moved. The corresponding pins are moved by moving the heads on the pins in an opposite direction relative to the body.

Thus, the heads714on the pins704are moved in a direction relative to the body702at the electronic device500, for example, providing a depression in the surface that generally follows the contour and profile of the object, such as a finger, that applied the force to the pins704. At the remote electronic device600, the heads on the corresponding pins are moved in an opposite direction relative to the body, for example, forming a projection that generally follows the contour and profile of the object that applied the force to the pins704. The projection formed at the remote electronic device600is formed by the heads on the pins, giving the general appearance of the object that applied the force to the heads714on the pins704at the electronic device500.

Thus, the shape formed by the movement of the heads714on the pins704relative to the body702when an external force is applied to the heads714at the electronic device500, is the inverse of the shape formed by the movement of the heads on the pins at the remote electronic device600. For example, a user pressing down with a hand on the heads714on the pins704presses with the palm toward the body702of the electronic device500. For the remote electronic device600in communication with the electronic device500, the shape that is formed follows the contours of the hand, with the palm of the hand facing away from the body of the remote electronic device600.

When external forces are applied to the heads714on the pins704at the electronic device500, and external forces are applied to heads on corresponding pins at the remote electronic device600, the heads on the corresponding pins at the remote electronic device600apply forces to the external object applying the forces at the remote electronic device600. Similarly, the heads714on the pins704at the electronic device500apply forces on the object applying the external force at the electronic device500. The forces applied by the pins at the remote electronic device600to the external object generally correspond in magnitude and direction to the external forces applied to the heads714on the pins704at the electronic device500. The forces applied by the heads714on the pins704to the external object at the electronic device500generally correspond in magnitude and direction to the external forces applied to the heads on the pins at the remote electronic device600.

Utilizing the movement of the heads714on the pins704and force application, the electronic device500, in cooperation with a remote electronic device600, simulates touch between two people that are each utilizing a respective one of the electronic devices. Utilizing such electronic devices500, touch contact is simulated to give the users the perception of touch.

As described above, software may be utilized to smooth out actions and reactions to generally maintain the simulation of touch and facilitate perception of touch by each user to compensate, at least in part, for latency introduced from various sources.

In addition to simulating touch, the heads714may optionally be utilized to emit audio. For example, the heads714may be moved together to collectively emit audio, similar to a speaker.

The heads714may optionally be operable to be heated or cooled or both heated and cooled, for example utilizing a heating or warming fluid within the pins704. Alternatively, a heating filament may be disposed within or around each head714or pin704. Utilizing a heating element or fluid, the heads714may be heated, for example to about the skin temperature of the sender. In addition, a thermocouple may be included in the pins704or in the heads714to measure the temperature of the pins704or the heads714.

In addition to detecting forces and to simulating forces or objects applied to a remote electronic device, the heads714on the pins704or the flexible, elastic membrane may be utilized to detect touches. For example, a patterned layer or layers of indium tin oxide may be deposited on the surface of the heads or on the surface of the elastic membrane for detecting touches thereon. For example, sensors may be disposed on or near the outer surface of the electronic device500for mutual-capacitance touch sensing.

Capacitive touch sensors may be used independently or in conjunction with other sensors to obtain input, for example to identify external contact with the device. For example, capacitive touch sensors may be used to distinguish between input that is a result of contact with skin, which is sensed utilizing capacitive sensors, or with a non-conductive object, which is not sensed utilizing capacitive sensors. The signals provided to the remote electronic device600may include such information to alter the tactile sensations provided to the user of the remote device600. For example, in response to determining that the input is a result of contact with the skin, signals are sent to cause heating of the interface to simulate skin contact. In response to determining that the input is not a result of skin contact, the signals sent to the remote electronic device600do not result in heating of the interface. The recipient at the remote electronic device600may alter the touch interaction to scale the simulated contact, for example amplifying or reducing the force, to change the temperature, or to make any other suitable modification.

In another aspect, sensors, including capacitive touch sensors or proximity sensors may be utilized to modify, turn on, or turn off data transmission, reception, or implementation. For example, the electronic device500may be utilized to transmit signals to the remote electronic device600when the electronic device500is not being held up to the ear of the user. The electronic device may also modify the touch data or discontinue sending signals that result from the user holding the electronic device500being held up to the user's ear, such as for voice communication.

Referring toFIG. 7A, pores718in the body702are distributed generally evenly across the surface of the body702, between the pins714. Alternatively, pores may be concentrated in specific areas. The speaker524shown inFIG. 5may be located in the body702to output audible information through the pores718and thus, at least some of the pores718are utilized as audio channels. Similarly, the microphone may be located in the body702to receive audible information through the pores.

The pores718shown inFIG. 7Aare pores in the surface of the body702. Alternatively, pins that include the pores may extend from the body702. In the example in which the pores are included in the pins, the pins that include the pores do not include heads714. For example, the pins may extend in between four heads714such that the pore is disposed between the four heads714. Such pores may be utilized as audio channels. Alternatively, the pins that do not include heads714may be utilized as fluid conduits to express gas or liquid therefrom. The gas or liquid may be disposed in one or more reservoirs disposed in the body702and expressed via one or more of the pins that do not include heads714.

Optionally, pores720may be disposed in some of the heads714, as illustrated inFIG. 7G. Some of the pores718are utilized to expel fluid, such as water, or to expel gas, such as air or to create suction. Thus, these pores718may be in communication with a reservoir, for example, to expel gas or liquid therefrom or in communication with a vessel to create a pressure difference to cause suction through the pores. Multiple pores may be disposed in each head714of at least some of the heads714to carry out various functions simultaneously.

In the example described above with reference toFIG. 7, a head714is disposed on each pin704. Rather than pores718disposed in the body702of the electronic device500, the pores may be disposed in the head714on the pins704. Optionally, the pins704may have a hollow section or fluid conduit722in communication with a reservoir724for the passage of fluid through at least part of the pins704and through pores in the head714.

Such pores may also be utilized for cleaning. For example, a cleaning fluid may, optionally be loaded into the device and expelled through the pores for cleaning the heads714on the pins704and the body702. In this example, the pins704may move such that the heads714move relative to the body702in more than one direction to distribute the cleaning fluid across the electronic device and for self-cleaning.

According to another example embodiment, the pins704are covered by a flexible, elastic membrane812such as a latex, flexible PVC, CyberSkin® or a combination of flexible, elastic materials, as illustrated inFIG. 8AandFIG. 8B. In this example, the elastic membrane may include the pores820in communication with one or more hollow passages822or fluid conduits in the pins704. The hollow passages822may be in fluid communication with one or more reservoirs, such as a liquid reservoir824and a gas reservoir826.

Alternatively, the flexible elastic membrane may be coupled to the pins704and fluid may be pumped into areas in the flexible elastic membrane or a reservoir below the flexible elastic membrane to inflate the flexible elastic membrane, for example, to fill in areas between pins. The fluid may be warmed or cooled such that the fluid provides heat or is cool to the touch for improved simulation of touch.

Utilizing such pores820, air or gas may be expelled, for example, to simulate blowing of air, and air may be sucked inwardly to create suction, for example, to simulate a kiss when a user at a remote device has his or her lips on the remote electronic device. The pores may also be utilized to detect when a person blows air onto the electronic device500, by detecting changes in air pressure, sound, or both air pressure and sound, and pores at the remote electronic device may be utilized to expel air. The pores may also be utilized for the passage of sound, or light. Optionally, fine hair or hair-like material may be moved through the pores to simulate fine wisps of hair on human skin, for example.

As indicated, the pores may also be utilized for cleaning. The cleaning fluid may be expelled through the pores for cleaning the elastic membrane812. The pins704may move relative to the body702to distribute the cleaning fluid across the membrane. Alternatively or in addition, the elastic membrane812, may be wiped clean by the user.

In the above-described embodiments, the pins704are generally evenly distributed in an array across the body702and extend from the body702. The pins704may be different sizes and may include different articulating joints or other couplings. For example, the pins704may be disposed on the body702such that the heads714of the pins704are disposed in different layers relative to the body702. The pins704may be offset from each other but are disposed at different distances from the body. Thus, a pin may extend a greater distance from the body702than an adjacent pin. In one example, three layers of pins may be disposed on the body702. The use of different layers of pins facilitates movement at greater depths, for example, for simulating a handshake or a hug. The stacked heads714also facilitate movement of the heads to cause a change in volume, for example, as heads move around from a stacked position to project outwardly, laterally or otherwise. The heads714on the pins may also be generally stacked on each other on the body702.

As indicated above, the heads714may include displays528embedded therein or disposed thereon to display an image or images on the heads714on the pins704. Images may be displayed on sides of the heads714as well as a top. When the pins are stacked, the images on the tops of the heads714and on the sides of the heads714provide depth to the image. The heads714or portions of the heads714may also be transparent such that an image or images are displayable through the heads714. Each head714may include a single pixel or a plurality of pixels, similar to pixels of a liquid crystal display (LCD), for example. Together, the pixels on the heads714are utilized to display information, such as an image. Thus, the controller516and the main processor502may be utilized to identify the location of each of the heads714and to coordinate the color and brightness of the pixels of the heads714to provide the image.

Alternatively, a flexible display may be utilized on the pins such that the pins cause movement and flexing of the display. In this example, the display is disposed on the pins and is operable to display information such as images.

Alternatively, the heads714may be a set color. Rather than displays incorporated into the pins704or heads714, images may be projected onto the heads714. Images may also be displayed on the sides of the pins, to the extent that the sides of the pins are exposed. In this case, the pins themselves include displays embedded therein or disposed thereon. Alternatively, images may be projected onto the pins.

The pins may include optical fibers or similar elements that transmit visual data through the pins. Such fiber optic or similar elements may be utilized in conjunction with pin positioning to provide image depth.

Optionally, the cameras530may be utilized to obtain images or video of the user's surroundings as well. Each of the cameras includes the functional components for operation of the camera. Video or images from multiple cameras may be combined by programmatically stitching together the video or images.

The cameras530may be utilized to obtain images of video of the user and part of the user at which contact occurs. The images or video may be provided generally in real time or near real time.

Additionally, the pins704are movable relative to the body702. The pins704are movable along the body702in a sliding or gliding motion, and movable toward and away from the body702. Different portions of the pins704are also moveable relative to the body702to facilitate movement of the pins704in other directions. The pins704may be moved closer together or farther apart on the body702. For example, the heads714on the pins or ends of the pins may be moved closer together by the various actuators controlling the couplings.

Optionally, the pins704may be grouped such that groups of the pins704may move together relative to the body702. For example, the groups of pins704may be coupled to an intermediate seat or base that is coupled to the body702. Sets or clusters of pins may swivel or pivot together on the base, relative to the body702, about a point or axis. Sets or clusters of pins may also move together with the base, away from the body702or toward the body702.

The movement pins704is controlled programmatically to facilitate the movement of individual pins704together as a group and to control the movement of sets of pins704together. Thus, for example, when a set of pins704move together on a base, relative to the body, other pins may move to accommodate the movement of the set of pins, such that the movement of pins704does not interfere with movement of other pins704.

The movement of groups of pins together also facilitates the simulation of more complex touch interactions in which forces are applied in more than one direction or plane, by comparison to the simulation of a surface or applied force in one direction.

When grouped together, the pins, along with the base on which the pins are disposed, may be removed and loaded on the body, for example, similar to the loading of a cartridge. Thus, pins that are worn or not working may be replaced by replacing a cartridge that includes a plurality of the pins.

Alternatively, the heads of the pins may be replaceable, for example, in the circumstance in which the heads wear out faster than the pins. The heads may be detachable or decouplable and the pins and heads programmatically controlled such that the heads are decoupled from the pins and new, replacement heads are coupled to the pins. For example, the pins may extend into a cartridge that includes replacement heads, where a replacement head is attached, and the pins are then retracted.

In addition, cartridges of pins may be selected based on the material or materials on the heads on the pins. For example, a cartridge may be selected to simulate a surface of a hand or to simulate clothing. Thus, rather than having different facets on the heads on the pins, cartridges of pins may be selected to simulate different surfaces.

The body may be any suitable size. As indicated above, the electronic device may be incorporated into a case for a smart phone. The body may also be much larger. For example, the body may be incorporated into a case or a part of a tablet computer. The body may be the size of a desk, small or large, or may be the size of a mattress. Two electronic devices in communication with each other may also be different sizes.

In the example ofFIG. 6, the electronic device500may be the size of a desk. The remote electronic device600, however, may be incorporated into a case for a smart phone. The electronic device may be configured to compensate for differences in size of the electronic device, for example, to fill in parts of an object for which signals or information is not transmitted.

A flowchart illustrating a method of controlling an electronic device, such as the electronic device500is shown inFIG. 9. The method may be carried out by software executed, for example, by the main processor102of the electronic device100. Coding of software for carrying out such a method is within the scope of a person of ordinary skill in the art given the present description. The method may contain additional or fewer processes than shown or described, and may be performed in a different order. Computer-readable code executable by at least one processor to perform the method may be stored in a computer-readable medium, such as a non-transitory computer-readable medium.

The method is carried out during a communication session with a remote electronic device, for example, at410or between408and410in the method ofFIG. 4. The signals are received at the local electronic device500, from the remote electronic device600, in response to externally applied forces that are detected at the remote electronic device600.

Based on the signals received, the object is identified902. For example, the signals received may be signals from fingers touching the remote electronic device. In this example, the electronic device500determines that the fingers extend to the edges of the interface and are part of a hand.

The electronic device500identifies, at904, a matching file stored in memory based on the identification of the object at902. The matching file includes information for providing signals to actuators518to simulate the fingers and hand of the user of the remote electronic device600. For example, the electronic device500may identify a specific user's hand based on identifying features including the shape and contours of the fingers. Alternatively, the electronic device500may identify a suitable hand by size and shape to go with the fingers identified at902.

In addition to actuating the actuators to simulate the portions of the object that touched the interface at the remote electronic device600, actuators518are actuated to simulate the missing parts of the object utilizing the file identified at904. Thus, in the example of the fingers touching the remote electronic device600, in addition to actuating the actuators518to simulate the fingers at the electronic device500, actuators518are actuated to control movement and forces applied by pins to simulate the hand.

Thus, the electronic device500is operable to add or fill in parts of objects. This method is particularly useful in the example in which the sizes of the electronic devices differ.

As indicated above, the body may be any suitable size. In addition, the body may take any suitable shape. For example, the body may envelop the user. Such a configuration is useful, for example, for simulating a hug or for virtual-reality applications. Other shapes may also be desirable, including a mattress, a chair, or other shape.

Signals sent to the remote electronic device600as a result of touch interaction with the electronic device500may also be scaled based on the size of each electronic device500and the remote electronic device600. For example, signals resulting from touch contact may alter the area of touch contact at the remote electronic device600. A ratio may be set automatically based on device sizes. For example, an 8″ electronic device in communication with a 4″ remote electronic device, may scale touch contact or movements or both by a factor of 2. Alternatively, scaling may be manually entered or may be determined based on predetermined rules. Alternatively, a smaller area of the electronic device500may be utilized such that the area of the interface of the electronic device500that is utilized is equivalent to the area of the interface of the remote electronic device600.

Optionally, sensors, such as an accelerometer or other suitable sensors, may be utilized detect movement of the electronic device, for example, when the entire electronic device500is being moved, for example, while a user is holding the device in a hand or hands. The electronic device500may also determine that no active movement is detected, for example, when the electronic device500is set down on a table and is stationary for a threshold period of time. A threshold force may be utilized to determine whether or not to send signals to the remote electronic device600. Different thresholds may be utilized to determine whether or not to transmit signals resulting from touch interaction depending on whether the electronic device500detects movement of the entire device or detects that the device is stationary. For example, an electronic device500that is stationary for 10 seconds, may utilize a higher threshold than when the electronic device500detects active movement, such that a greater force is required to cause the electronic device500to send signals to cause the remote electronic device600to apply a force.

Another example of an electronic device is shown inFIG. 10. In this example, a flexible elastic membrane1012is coupled to pins1004. The pins1004each include a fluid conduit1020that extends through the pin to a reservoir1022in the body1002. The fluid conduit1020is utilized for fluid communication between the reservoir1022and a respective pocket1024between layers of the flexible elastic membrane1012. The flexible elastic membrane includes a plurality of pockets1024for receiving fluid from the reservoir1022. The fluid may be a gas or a liquid or both gas and liquid. Thus, fluid may be pumped into the pockets1024to inflate the flexible elastic membrane1012. In this example, the pins1004are coupled to the flexible elastic membrane1012and the flexible elastic membrane1012is moved by increasing or decreasing the fluid in the pockets1024, thereby expanding or collapsing the pockets1024. When a pocket expands as fluid is pumped into the pocket, the outer surface of the flexible elastic membrane1012is moved outwardly, away from the body1002. When a pocket collapses, the outer surface of the flexible elastic membrane1012is moved inwardly, toward the body1002. Thus, the outer surface of the flexible elastic membrane1012is moveable relative to the body1002.

Actuators1018control fluid movement from the reservoir1022to the pockets1024. The actuators1018are utilized to cause the fluid to flow along the respective pins1004and thereby cause movement of the outer surface of the flexible elastic membrane1012relative to the body1002. Thus, each actuator is individually controllable to control the movement of parts of the outer surface of the flexible elastic membrane1012. The movement of the flexible elastic membrane1012is controlled to simulate touch contact. The fluid may be warmed or cooled, utilizing a heating element or a cooling fluid disposed in the area1026around the fluid conduit1020such that the fluid provides heat or is cool to the touch for improved simulation of touch.

Force sensors are also associated with the flexible elastic membrane1012, for example, to detect external forces applied to the flexible elastic membrane1012.

The method shown inFIG. 4and described herein is also applicable to the electronic device shown inFIG. 10. The method may be carried out by software executed, for example, by a main processor (not shown) of the electronic device. Details of the method shown inFIG. 4and described above are also applicable to the electronic device500and are therefore not described again herein.

Thus, during a communication session, externally applied forces on the interface of the local electronic device1000are detected and, in response, signals are transmitted to the remote electronic device. Signals are also received at the local electronic device1000in response to externally applied forces that are detected at the remote electronic device.

In response to receipt of signals at the local electronic device at408, the actuators1018are actuated at410to control movement of portions of the flexible elastic membrane1012to thereby control movement and forces applied by the flexible elastic membrane1012.

Because the flexible elastic membrane1012is movable toward and away from the body302, the flexible elastic membrane1012is operable to apply a force to an object touching the flexible elastic membrane1012. In addition, the flow of fluid into the pockets1024is controlled to form a shape, such as a projection, that generally follows the contours and surface profile of an object touching the interface of a remote device that is in communication with the electronic device1000. Utilizing the movement of the flexible elastic membrane1012and force application, the electronic device1000, in cooperation with a remote electronic device, simulates touch between two people that are each utilizing a respective one of the electronic devices.

Yet another example of an electronic device is shown inFIG. 11. In this example, a flexible elastic membrane1112is coupled to the body1102. The flexible elastic membrane1112includes a plurality of pockets1124for receiving fluid therein. Fluid conduits1120extend through an upper surface of the body1102to a reservoir1122in the body1102. The fluid conduits1120are utilized for fluid communication between the reservoir1122and respective pockets1124between layers of the flexible elastic membrane1112. The fluid may be a gas or a liquid or both gas and liquid. Thus, fluid may be pumped into the pockets1124to inflate the flexible elastic membrane1112. The flexible elastic membrane1112is moved by increasing or decreasing the fluid in the pockets1124, thereby expanding or collapsing the pockets1124. When a pocket expands as fluid is pumped into the pocket, the outer surface of the flexible elastic membrane1112is moved outwardly, away from the body1102. When a pocket collapses, the outer surface of the flexible elastic membrane1112is moved inwardly, toward the body1102. Thus, the outer surface of the flexible elastic membrane1112is moveable relative to the body1102.

A controller controls fluid movement from the reservoir1122to the pockets1124and from the pockets1124to the reservoir1122. The controller is utilized, in conjunction with valves, to cause the fluid to flow through the fluid conduits1020, which include apertures in a surface of the body1102and thereby cause movement of the outer surface of the flexible elastic membrane1112relative to the body1102. Thus, the controller, which may include valves, for example, controls the movement of parts of the outer surface of the flexible elastic membrane1112. The movement of the flexible elastic membrane1112is controlled to simulate touch contact. The fluid may be warmed or cooled such that the fluid provides heat or is cool to the touch for improved simulation of touch.

Force sensors are also associated with the flexible elastic membrane1112, for example, to detect external forces applied to the flexible elastic membrane1112.

The method shown inFIG. 4and described herein is also applicable to the electronic device shown inFIG. 11. The method may be carried out by software executed, for example, by a main processor (not shown) of the electronic device. Details of the method shown inFIG. 4and described above are also applicable to the electronic device500and are therefore not described again herein.

Thus, during a communication session, externally applied forces on the interface of the local electronic device1100are detected and, in response, signals are transmitted to the remote electronic device. Signals are also received at the local electronic device1100in response to externally applied forces that are detected at the remote electronic device.

In response to receipt of signals at the local electronic device at408, the controller controls movement of portions of the flexible elastic membrane1112to thereby control movement and forces applied by the flexible elastic membrane1112.

Because the flexible elastic membrane1112is movable toward and away from the body1102, the flexible elastic membrane1112is operable to apply a force to an object touching the flexible elastic membrane1112. In addition, the flow of fluid into the pockets1124is controlled to form a shape, such as a projection, that generally follows the contours and surface profile of an object touching the interface of a remote device that is in communication with the electronic device1100. Utilizing the movement of the flexible elastic membrane1112and force application, the electronic device1100, in cooperation with a remote electronic device, simulates touch between two people that are each utilizing a respective one of the electronic devices.

Referring toFIG. 12and as indicated above, pins1204may be disposed on a body1202such that the heads1214of the pins1204are disposed in different layers relative to the body1202. Thus, the heads1214on the pins are generally stacked on the body1202. Stacking of heads1214facilitates detection of forces and movement at greater depths, for example, for simulating a handshake or a hug. The stacked heads1214also facilitate movement of the heads to cause a change in volume, for example, as heads move around from a stacked position to project outwardly, laterally or otherwise. Images may be displayed on sides of the heads1214as well as a top. When the pins are stacked, the images on the tops of the heads1214and on the sides of the heads1214provide depth to the image.

The pins1204are grouped such that groups of the pins1204move together relative to the body1202. For example, outer groups of pins1230are disposed on intermediate groups of pins1232, which are disposed on inner groups of pins1234. In this example, the pins1204of the outer groups of pins1230are smaller than the pins1204of the intermediate groups of pins1232and the pins1204of the intermediate groups of pins1232are smaller than the pins1204of the inner groups of pins1232. Thus, a plurality of pins of an outer group of pins120is disposed on one of the intermediate pins1232. Similarly, a plurality of pins of an intermediate group of pins is disposed on one of the inner pins1234.

Movement of one of the inner groups of pins1234results in movement of the associated intermediate groups of pins1232and the associated outer groups of pins1230. The outer groups of pins1230, the intermediate groups of pins1232, and the inner groups of pins1234include respective couplings or joints facilitating movement of the pins in three dimensions. Thus, the groups, also referred to as clusters of pins may swivel or pivot together relative to the body120. The movement of the pins1204is controlled programmatically to facilitate the movement of individual pins1204together as a group and to control the movement of groups of pins1204together.

The movement of groups of pins facilitates the simulation of complex touch interactions. In addition, the pins1204may move to form complex shapes, such as the chair1300illustrated inFIG. 13.

The described embodiments are to be considered as illustrative and not restrictive. The scope of the claims should not be limited by the preferred embodiments set forth in the examples, but should be given the broadest interpretation consistent with the description as a whole. All changes that come with meaning and range of equivalency of the claims are to be embraced within their scope.