Approaches for device location and communication

An electronic device can utilize image capture technology to detect the presence and location of another device. Using this information, the electronic device can display, in a user interface, a graphical element representing a detected device, along with identity information and the location of the detected device relative to the electronic device. The location of each detected device relative to the electronic device can be tracked and thus the graphical element can be updated in the user interface.

BACKGROUND

The number of people utilizing portable computing devices to communicate with each other, as well as the number of available computing devices, continues to increase. Today, in order to deliver a document or any other type of information electronically via a portable computing device to another person requires the sender to know certain information about the recipient, such as an email address, phone number or network address. Alternatively, a user may load the document onto a flash drive or other storage medium and provide the storage medium to the other user, which requires the other user to ignore possible security risks and load the document from the storage medium onto another device.

Certain technologies such as Bluetooth® and wireless networking (e.g., Wi-Fi) enable users to share information wirelessly. However, these wireless technologies still require a sender to obtain an identifier associated with the recipient or the recipient's device that the sender would like to share information with. For example, in some instances, the sender's device may display a list of available recipient devices that can be selected in order to establish a connection between the sender's device and the recipient's device. The list of available devices typically does not provide any information other than a proxy name associated with the recipient device. Thus, in a room of multiple people, it can be difficult (if possible at all) to determine which device belongs to which person in the room simply by looking at the list of available devices.

DETAILED DESCRIPTION

The technology presented herein describes an electronic device that may detect the presence of other devices based on captured images. When other devices are detected, the electronic device can determine a location of each of the other devices relative to the electronic device and determine an ability to communicate with each of the detected devices. Using this information, the electronic device can display a graphical element representing the detected device in a user interface. The graphical element can be displayed in the user interface in a location representing the location of the detected device relative to the electronic device. The graphical element may be accompanied by at least one of identity information associated with the detected device and an ability to communicate with the detected device.

Various other applications, functions, and advantages are presented below with respect to the various embodiments. It should be understood that the description and figures provide a number of examples, but the alternatives and variations possible within the scope of the various embodiments are not fully described. These alternatives and variations, however, would be apparent to one of ordinary skill in the art in light of the teachings and suggestions contained herein. It should be understood that reference numbers are carried over between figures to identify similar elements for purposes of simplicity of explanation alone, and this should not be interpreted as limiting the scope of the various embodiments or showing various options as part of the same embodiment.

FIG. 1illustrates an example of a portable computing device100that can be used with various embodiments discussed herein. For purposes of explanation, the primary portable electronic device described herein will be referred to as a “client device” that is able to detect the presence of, and communicate with, other nearby devices. The example client device100includes conventional computing elements, such as a processor and memory, as discussed in more detail below with regard toFIG. 2, as well as image capture elements102and a display element104. The client device100may also include one or more orientation detection elements106, a location detection element107, a housing108and a microphone110. The display element104can comprise a display screen, touch-sensitive or otherwise, which can display various graphical elements as part of a graphical user interface (GUI). A GUI can enable the user of the client device to input various characters or text, select various elements or commands or provide another such input, using various approaches discussed herein and known in the art for such purposes.

The orientation detection element(s)106(e.g., an accelerometer or gyro element) can be used to determine orientation, motion or other movement or changes in orientation of the client device100. The client device can also include at least one other conventional input mechanism (not shown), such as one or more user input elements (e.g., buttons, pressure-sensitive surface), positioned at any appropriate location on the client device, such as the front surface as in conventional devices (e.g., as part of a keyboard or keypad) or the side or back surface of the device such that a user holding the client device would likely be able to press or otherwise activate the button(s) without having to substantially shift hand positions. Various other input elements can be used as well, such as an audio input using a microphone element110, etc.

Image capture elements102capture one or more images that may be used to detect the presence of another device, and in some embodiments, determine the location of a detected device relative to the client device. The example client device100shown inFIG. 1includes six image capture elements102, with each of the image capture elements102(1)-102(4) located on each of the four sides of the device, image capture element102(5) located on the front side or top of the device, and an image capture element on the back side or bottom of the device (not shown). In some embodiments, an image capture element102may be a charge coupled device (CCD), a digital still camera or a video camera and may include a lens for altering the field of view of the image capture element102. In other embodiments, an image capture element102may be an infrared element.

Each of the image capture elements102shown inFIG. 1has a field of view θ. As shown inFIG. 1, image capture element102(1) has a field of view θ1; image capture element102(2) has a field of view θ2; image capture element102(3) has a field of view θ3; and image capture element102(4) has a field of view θ4. The field of view for each image capture element102may be the same or have a varying field of view. Image capture elements102(1)-102(4) enables the client device100to capture one or more images in substantially any direction from the client device without having to move or rotate the client device. The configuration of image capture elements102shown inFIG. 1is for example only. Image capture elements102can be positioned on the client device in other locations that are least likely to interfere with the user's comfortable operation of the device.

The location detection element107can be used to determine the location of a detected device relative to the client device. In some embodiments, a location detection element107may be an infrared (IR) transceiver. In other embodiments, a location detection element107may be a GPS or ultra-wide band (UWB) transceiver. Regardless of the type of transceiver, the location detection element107is used to determine a location of a detected device relative to the client device. In instances where the client device and at least one detected device each include a UWB transceiver, a UWB technique may be used to determine the location of the detected device relative to the client device. UWB uses round-trip time of flight measurements for estimating the location of another device relative to the client device. This UWB technique, which is known in the art, measures the total elapsed time for a two-packet exchange between the client device and a detected device.

FIG. 2illustrates example components of a computing device200, such as the client device100shown inFIG. 1. In this example, the device includes a processor202for executing instructions that can be stored in a memory device or element204. As is known in the art, the device can include many types of memory, data storage or computer-readable media, such as data storage for program instructions for execution by the processor202, a separate storage for images or data, a removable memory for sharing information with other devices, etc. The device typically will include some type of display element206, such as a touch screen or liquid crystal display (LCD), although devices such as portable media players might convey information via other means, such as through audio speakers. In addition, the device may acquire many different types of information (e.g., location, presence, identity etc.) and provide this information to another device with a display element.

An image capture element208may be, by way of example only, an ambient light camera, infrared (IR) transceiver or any combination thereof. An image capture element can include any appropriate optical element useful in providing sufficient angular spread, focal range and viewable area to capture one or more images of a user of the device, an intended target and/or devices positioned around the client device within a given range. Methods for capturing images using one or more image capture elements with a computing device are well known in the art and will not be discussed herein in detail. It should be understood that image capture can be performed using a single image, multiple images, periodic imaging, continuous image capturing, image streaming, etc. Further, a device can include the ability to start and/or stop image capture, such as when receiving a command from a user, application, or other device.

Orientation detection element210is able to determine and/or detect orientation and/or movement of the device200. Methods for using elements such as accelerometers to determine orientation or movement of a device are also known in the art and will not be discussed herein in detail. A location detection element211may be, by way of example only, an image capture element, a UWB transceiver, a GPS transceiver, an IR transceiver or any combination thereof. In some embodiments, the device can include at least one additional input element212able to receive input from a user. An input element can include, for example, a push button, touch pad, touch screen, touch-sensitive material, wheel, joystick, keyboard, mouse, keypad or any other such device or element whereby a user can input a command to the device. A client device may also include a microphone or other audio capture element (shown as microphone110inFIG. 1) that accepts voice or other audio commands. In this case, a device could be controlled through a combination of visual and audio commands, such that a user can control the device without having to physically contact the device.

FIG. 3illustrates a configuration300wherein a client device310is nearby four other computing or electronic devices: an electronic book reader302(also referred to herein as a reader), a notebook computer304, a cellular phone306and a portable data assistant (PDA)308. As described above, the image capturing elements of the client device may continually capture images, periodically capture images, etc. The configuration shown inFIG. 3, which by way of example only, represents the client device310being held by a user and located within the same room as the other devices302,304,306,308.

FIG. 3illustrates that each image capture element has a field of view θ. Image capture element312has a field of view θ1. Image capture element314has a field of view θ2. Image capture element316has a field of view θ3. Image capture element318also has a field of view, but the field of view is not visible from the perspective of the client device shown inFIG. 3.FIG. 3further illustrates that, in some embodiments, the field of view of an image capture element may overlap with another image capture element. For example, field of view θ1overlaps with field of view θ2at some distance from the device. Similarly, field of view θ1overlaps with field of view θ3at some distance from the device. It is not required that the image capture elements have overlapping fields of view.

Client device310may also detect the presence of another nearby device and determine the physical location of each nearby device based at least in part on the image(s) captured by the image capture elements. Using the configuration300shown inFIG. 3, the client device310, through image capture elements312and314, may capture at least a portion of the cellular phone306in an image because cellular phone306is within the field of view of both the image capture elements312and314. Similarly, image capture element312and image capture element316may capture at least a portion of the PDA308in an image because the PDA308is within the field of view of both image capture elements312,316. Depending upon the tilt angle of the client device, the image capture element326or an image capture element located on a back surface of the device310(not shown) might also capture at least a portion of the cellular phone306and/or PDA308. In some embodiments, the client device itself is able to detect the presence of a nearby device (e.g., cellular phone306) when at least a portion of a nearby device is present in a captured image. In other embodiments, the client device delivers the one or more captured images to a remote service that is able to detect the presence of a device based on the captured image(s). In this instance, the client device may receive a response from the remote service indicating whether a device is present in the one or more captured images. Any image recognition technology known in the art today may be used for detecting the presence of a device in a captured image.

Upon detecting the presence of another device in a captured image, an algorithm executing on the client device310(or remote to the client device) can determine an approximate location of each detected device relative to the client device310using one or more captured images, IR, UWB or any combination thereof. Suppose that each image capture element312shown inFIG. 3is, for example, a camera that may capture one or more images that includes at least a portion of reader302, laptop304, cellular phone306and PDA308. Based at least in part upon the image captured by image capture element312, the client device310(or remote service) can determine an approximate direction of each of the devices302-308with respect to the client device310.

Client device310(or remote service) can, in some embodiments, approximate a distance from the client device310to each of the detected devices302-308. For example, the client device310may include a first algorithm that determines the distance of the electronic book reader302from the client device310based on the size of the book reader in the one or more captured images. In some embodiments, the client device310can deliver the one or more captured images to an image recognition service (or other such source) to determine a likely type and/or model of the reader302. In some embodiments, information identifying a model or type of device might be contained in a label on the device or sent as part of a transmission from the device. Based at least in part upon this information, an algorithm can correlate the size of the particular model of the device, for example, with the apparent size of the device in the image, in order to more accurately determine a distance from the client device310to the reader302.

The distance from the client device310to another device may also be determined using more than one captured image. For example, in embodiments where at least two image capture elements capture an image showing at least a portion of the reader302, the client device310(or remote service) can analyze both images in order to attempt to perform triangulation or another similar process for determining the location of the reader302relative to the client device310. For example, the client device310can analyze a first image from the first image capture element312that includes at least a portion of the cellular phone306and a second image from the image capture element314also including at least a portion of the cellular phone306. Upon analyzing the first and second images, the client device310(or remote service) can determine a relative size of the cellular phone306and/or location of the cellular phone306relative to the client device310. In some embodiments, the client device can also determine the approximate distance from the client device to the cellular phone306.

FIG. 3illustrates that the client device310can present to a user, via a display element328, a position of detected computing devices relative to the client device310. InFIG. 3, an icon or image is displayed or presented in or on the display element328representing each detected device (e.g., cellular phone306, reader302). In some embodiments, a client device icon330is also displayed on display element328. For example, inFIG. 3, a cellular phone icon322is displayed on display element328relative to the client device icon330representing the location of the cellular phone306relative to the client device310. Similarly, an electronic book reader icon318is displayed on display element328relative to the client device icon330representing the location of the reader302relative to the client device310. A notebook icon320is also displayed on display328relative to the client device icon330representing the location of the notebook304relative to the client device310. Finally, a PDA icon324is displayed on display328relative to the client device icon330representing the location of the PDA308relative to the client device310. In some embodiments, the icons visible on display328may be actual images of the various detected devices (e.g., retrieved from a captured image). In other embodiments, when the type of device may be determined, the icons visible on display328may include stock images of the detected device, as may be retrieved from a central data store or other such location from across a network or stored on the client device.

In some embodiments, one or more of the image capture devices of the client device are, or also include, an infrared (IR) transceiver to emit and receive focused infrared radiation for short-range communication. The client device310may also include one or more IR transceivers in addition to the image capture elements. The IR transceiver can conform to an appropriate transmission standard, such as the standards set forth by the Infrared Data Association (IrDA). The client device can include at least one optical modulator operable to modulate the infrared beam from at least one IR transceiver in order to encode data within the generated beam. For example, a device number may be encoded inside the IR signal transmitted by a device. As will be discussed in more detail later, the client device would then be able to identify the detected device based on the encoded IR signal transmitted by the detected device. The client device can also include at least one photodiode or similar element operable to receive transmitted IR radiation, which can be demodulated in order to extract the information encoded into the received signal.

In the instance where the client device310has at least one infrared receiver (or transceiver) capable of receiving a transmission from an infrared transmitter of another device (e.g., cellular phone306), the client device310can determine an ability to communicate between the client device310and the other detected device(s). It should be understood, of course, that infrared signals are used for purposes of explanation only, and that any other appropriate type of signal can be used in accordance with the various embodiments.FIG. 3illustrates an example whereby the client device310, either as part of the initial device detection process or as part of a separate process, has determined that the client device310has an ability to communicate with the electronic book reader302, notebook computer304and cellular phone306. A user of the client device may visibly confirm that the client device310has an ability to communicate with each of these devices because the reader icon318, laptop icon320and the cellular phone icon322are shown on display328with solid lines.FIG. 3also illustrates that the client device310has determined that it has no ability to communicate with the PDA308. Again, a user of the client device310can visibly confirm that the client device310cannot communicate with the PDA308because the PDA icon324is shown in hidden lines on display328. The communication state of another device may be displayed in other ways such as, but not limited to, a color, highlighted, animated or any other state that would allow the display328to differentiate devices that may communicate with the client device as opposed to devices that cannot communicate with the client device.

In some embodiments, a user can provide certain other users with a code that the other users can enter into their respective devices. This code then can be broadcast with each communication from the devices having that code, such that only certain devices can communicate with each other even though multiple other devices might receive the transmissions, as the other devices would not have the code needed to decode the information in the transmissions. Any other code/key value may be broadcast with each communication to provide a secure handshake between devices. The code can be provided via any approach, such as email, instant messaging, social networking, file transfer or voice communication.

As shown inFIG. 3, a user of the client device may select any one of the icons318,320,322to initiate a communication with the particular device. In some embodiments, a user may also select one of the icons318,320,322to prevent communications with the particular device.FIG. 4(a) illustrates an example display state400wherein the communication state of each device is displayed adjacent the device icon. Icon320indicates that the laptop304is available for communication with the client device. Icon322indicates that the cellular phone306is available for communication with the client device. Icon324indicates that the PDA308is not available for communication with the client device.FIG. 4(a) shows that a user has moved a selection element (e.g., a cursor) to select the icon318corresponding to the electronic book reader302in order to attempt to establish a connection between the client device and the reader302. In some embodiments, the user could touch the icon318or perform another such selection action to establish communication between the client device and the reader302.

FIG. 4(a) illustrates that the client device can display a connection state402for the selected icon, shown inFIG. 4(a) as “connecting,” or any other similar message or indicator. Similar communication state information can be displayed along with the icons representing the other devices as well, such as an “available” state for the notebook icon320and cell phone icon322, which represents that those devices are able to communicate with the client device, but do not yet have a connection established. A “no signal” state is displayed with the PDA icon324, which indicates that the client device has not yet been able to determine an ability to communicate with the PDA308. A user may also intentionally not want to establish communications with another device. In such a case, a user may perform an action (e.g., right click, touch or perform a touch gesture such as a tap or a swipe) on an icon visible on display328for a particular device and selecting an appropriate menu option. For example, the user may drag an icon visible on display328to a trash bin icon (not shown) or by adding an identifier associated with the selected device to a blacklist stored on the client device (or remote service). By doing so, the client device can prevent communication with the particular device.

In some embodiments, the client device emits a signal, via the infrared emitter (or other such communication element), toward the device represented by the icon selected by the user. The signal emitted by the client device may correspond to a “wake up” code, while in other embodiments the signal could correspond to a “please acknowledge” or “please identify yourself” code. Any appropriate encoding approach can be used as known or subsequently developed for transmitting information over a signal such as an infrared signal. When the device corresponding to the selected icon318(electronic book reader302) receives the coded signal, the reader302can decode the signal and determine the contents of the request. In some embodiments, the reader302can automatically respond with an “acknowledge” message (or other similar code). In other embodiments, the reader302will prompt a user to first confirm the request or otherwise approve the sending of a response from the reader302back to the client device310.

It is possible that the signal transmitted by the client device310may be received by nearby devices other than the device the user is intending to establish communications with (e.g., laptop304, cellular phone306, PDA308, which are located in the same area as reader302inFIG. 3). In some embodiments, therefore, each electronic device responding to the request transmitted by the client device310can send identifying information along with the signal so that the client device310may be able to distinguish responses received from various devices. For example, returning toFIG. 3, if image capture element312is, or includes, an IR transceiver, a signal emitted from the image capture element312might be received by the cellular phone306and the electronic book reader302because both devices are within the field of view of the image capture element312. When the client device310receives a response from the device associated with the selected icon318, the communication state shown on display328can be updated, such as is illustrated in the example display state430ofFIG. 4(b). In this example, the icon318for the selected device has changed in appearance (e.g., color or size) and/or updated communication state information432is displayed, indicating that a handshake or other connection action occurred between the client device310and the reader302.

Approaches in accordance with various embodiments rely on some type of identifier, such as a client identifier, user identifier, etc., included within the responses received from the detected electronic devices responding to requests transmitted by the client device. By doing so, the client device can decode the information received from other devices (e.g., reader302) it intends to establish a communication connection based at least in part upon the direction from which the signal was received by the client device. The client device can therefore associate the location of another device relative to itself and some identifying information associated with the other device.

In some embodiments, a detected device (e.g., book reader302) can also automatically provide the client device310with information associated with a user of the detected device such as, but not limited to, the user currently logged into the detected device or the owner of the detected device. For example, the detected device can transfer to the client device information such as a name, an employer, an occupation or any other information that might be of interest to a user. It should be understood that any information can be transferred to the client device such as a gamer tag, social networking information and the like. This type of information can be sent to the client device as part of determining an ability to communicate with the other device, during a subsequent exchange of information between the client device and the detected device, or at any other time.

In other embodiments, the initial identifier received by the client device from the detected device (e.g., a user identifier or device identifier sent by the reader302) can be used to obtain additional user information. Again, using theFIG. 3example, an identifier received by the client device from the reader302can be referenced against contact information stored on the client device. In another example, the identifier received by the client device from the reader302can be delivered from the client device to a remote service that stores various types of information (e.g., address book, social network). Any additional information identified by the remote service that is related to the identifier may be returned to the client device for presentation to the user. In some embodiments, the identifier received by the client device from the reader302can correspond to a uniform resource locator (URL) identifying a location where information associated with the reader302can be obtained. In other embodiments, the identifier received by the client device from the reader302might include information that can be used by the client device to locate user information associated with the reader (e.g., by performing a search on the Internet). Various other sources of identity information can be used as well as should be apparent.

The identity information obtained by the client device, whether via itself or through a remote service, can be displayed (or be made available for display) to a user via the display328of the client device.FIG. 4(c) illustrates an example display state460wherein identity information associated with the reader302, the laptop304and the cellular phone306have been obtained and displayed. In particular,FIG. 4(c) shows that user identity information associated with each device that the client device could establish communication with, the reader302, the laptop304and the cellular phone306, was obtained and is being displayed on display328. As discussed above, various types of identity information associated with a device may be obtained. For reader302and laptop304, the device icons318and320shown inFIG. 4(b) were replaced with user images or icons462,466. In some embodiments, the user images462,466may be an actual image of the user associated with the device. In other embodiments, the user images462466are stock “male” and “female” images. Of course, any type of image may be displayed (e.g., user's avatar). These personalized images can help a user of the client device confirm that the correct device was detected by the client device and may also help the user of the client device identify people in a meeting, social gathering and the like. The images can be determined using any of the identity-determining approaches discussed above. If a user image cannot be associated with a detected device, an image of the particular device (or other appropriate image) can be displayed, or the device can continue to show the original icon468as illustrated for the cell phone306inFIG. 4(c).

The client device can also display additional identity information associated with the user of the detected device. For example, user image462representing the reader302is displayed along with identity information464, which in this example includes the user's name (“John Smith”), his company (“ACME Corp.”) and his position (“President”). Similarly, user image466representing the laptop304is displayed along with identity information467, which includes the user's name (“Mary Lee”), her company (“Products Intl.”) and her position (“Owner”). In the example shown inFIG. 4(c), client device310could not identify or locate a user image associated with cellular phone306. Thus, phone icon468is displayed representing the phone306. Even though a user image could not be displayed, identity information470associated with the cellular phone306was identified and is displayed, which includes the user's name (“James Jones”), his company (“Foo Inc.”) and his position (“CEO”). As discussed above, the identity information464,467,470displayed along with icons462,466and468may have been obtained automatically by the client device itself, through a remote service or a combination thereof. Identity information may also be manually input by a user into the client device. Using the example shown inFIG. 4(c), in the event that the client device (or remote service) could not obtain any identity information associated with the reader302, the user of the client device could manually enter identity information464into an address book, contact list, etc. of the client device or a remote service accessible by the client device so that identity information464can be displayed as shown inFIG. 4(c).

Other types of information associated with a particular device may be viewed as well. A user of the client device may customize the types of information displayed along with a device icon or user icon representing a detected device. For example, using the example shown inFIG. 4(c), a user, via the client device, may also view information such as how much business the user (or the user's company) has received from Foo Inc., Foo Inc.'s sales information, etc. In another example, a user in a business meeting might choose for the client device to show the company and title associated with the user of the device, as illustrated in the example information464displayed along with the icon462representing the user of the reader302. In a social setting, however, a user of a client device might prefer that the client device show other types of information along with a device icon or user icon such as a user's hobbies, family members, mutual friends and the like.

A limited amount of identity information may be visible at one time in display328of the client device310due to the size limitations of display328. In some embodiments, a device icon or user icon may be selected to obtain additional information about a particular user. Using the example provided inFIG. 4(c), the user icon462may be selected to view additional information about John Smith. In some embodiments, a pop-up screen or window will appear on display328of the client device310upon selection of the user icon462. The pop-up screen may include any appropriate information for that identified user that may be obtained by, or delivered to, the client device such as a user's personal bio, performance history, past communications with the user or list of recent articles written by or about the user. A user may have the ability to prevent others from obtaining certain types of non-public information about them. For example, a user may designate certain information as “public” information that anyone may view while other types of information are “private” information that may be accessed only by others with permission of the user. A user may also prevent any type of identity information from being obtained by a particular device. For example, if a user's client device detects a device associated with a competitor, the user can select the competitor's device or user icon visible on the display and select a “block information” option that would prevent the competitor's device from obtaining information about the user.

In order for the client device to properly locate and identify the devices associated with these users, as discussed above, the client device needs to be able to image or otherwise obtain information about the devices. As discussed above, this can involve a client device having one or more image capture elements positioned at appropriate locations on the device. Depending at least in part upon the type of image capture element and the intended use, there can be a number of potential variations and configurations used to capture information about nearby devices. As discussed previously with respect toFIG. 3, there can be an imaging element or camera on one or more sides of the client device, such that a substantially rectangular device can have six image capture elements/cameras in certain embodiments, although there could be more than one image capture element on a side or there can be at least one side that does not include an image capture element. For example, a “lower” edge of the device (e.g., the edge facing the user when the user is operating the device) might not include an image capture element in certain embodiments as the image capture element from that side might be unlikely to pick up devices that were not able to be captured by an element on one of the adjacent sides.

FIGS. 5(a)-5(d) illustrate a client device with additional configurations of image capture elements.FIG. 5(a) illustrates a client device500including four image capture elements502(1)-502(4). Each image capture element502is positioned on one of the corners of the client device500. Image capture element502(1) has a field of view θ4. Image capture element502(2) has a field of view θ5. Image capture element502(3) has a field of view θ6. Image capture element502(4) has a field of view θ7. Similar to the image capture elements shown inFIG. 3, each image capture element502may have the same, or varying, field of view. Even though the field of view shown inFIG. 5(a) for each image capture element is less than 180 degrees, in some embodiments, one or more of the image capture elements may be a wide angles lens (e.g., fisheye lenses) that has a field of view of 180 degrees or more.

FIG. 5(b) illustrates a client device530with a single image capture element532having a field of view θ8. A client device with a single image capture element will capture images only within the field of view of the one image capture element. Thus, if image capture element532was located on a front surface of the client device530(as shown inFIG. 5(b)), a user may be required to point the image capture element532in other directions, or move the client device left or right, to capture images of areas other than the area directly in front of the client device at the moment. However, a client device with a single image capture element can be less complex and less costly than a client device with multiple image capture elements. This embodiment may be used, for example, in a classroom environment where the teacher and teacher's device are positioned at the front of the room, and the students are all seated in front of the device.

FIG. 5(c) illustrates another alternative configuration of a client device. Client device560includes a single image capture element562positioned on a front face564of the client device560. In some embodiments, image capture element562may capture images only of the area in front of the client device (e.g., towards a user holding the client device560with the display facing the user). In other embodiments, the image capture element562is able to capture a panoramic image (e.g., a 360° view). The image capture element562can contain more than one lens or optical element in some embodiments in order to improve viewing angle, minimize optical defects or artifacts due to the wide viewing angle or perform a similar function.

FIG. 5(d) illustrates another configuration of a client device. Client device590includes an imaging element592and four lenses594(1)-594(4). Similar to the image capture elements312,314,316shown inFIG. 3, each lens594in the client device590may have a similar or varying field of view. Each lens594is connected to the imaging element592by, for example, a light tube596. A light tube596may be, by way of example only, a fiber optic cable, a tube lined with highly reflective material or any other device or element known in the art for transporting light/signals from one location to another. Each light tube596is used for transporting or distributing light entering a lens594to the imaging element592. In some embodiments, a lens594is a shutter device that allows light to pass through the lens for a determined period of time, for the purpose of exposing the imaging sensor to light to capture an image. In this instance, the client device590may be able to control and coordinate the operation of each lens594for capturing one or more images. One advantage of client device590is reduced cost of a single imaging element.

FIG. 6(a) illustrates an example client device602that includes an IR communication element604with a limited angular transmission range. As shown inFIG. 6(a), client device602has detected reader606, cellular phone610and PDA612, and icons618,620and622are visible on the display of client device602. In the example configuration600shown inFIG. 6(a), the client device602may communicate with a single device, for example device606, upon the user pointing the communication element604at a current location of device606. One advantage to such an approach for communicating between devices is that the other devices610,612nearby the client device602will not also receive the transmission from the client device602to the selected device606. If the user of the client device wants to share information with the other devices610,612, the user can point the communication element604of the client device602at each of the devices610,612and resend the transmission. Such an approach (limiting communication to a single nearby device) can provide an additional level of security, but can introduce some redundancy when information is to be shared among several devices. If the client device includes more than one IR transceiver, upon selection of icon618, information provided by one or more of the accelerometers in the client device may be used to determine which one of the IR transceivers should be used to transmit information to the device606.

FIG. 6(b) illustrates an example client device650including an image capture element652that is able to scan a viewing area656with a limited angular field of view654. The image capture element652may capture one or more images (e.g., digital images) each covering a portion of the viewing area656. Using such a configuration, the client device650can provide a “detection” mode wherein the image capture element652periodically (or continually) scans the area656in order to detect nearby electronic devices.

Image capture element652, upon scanning area656, may capture one or more images as the element652moves from left to right (from the perspective shown inFIG. 6(b)). Using the example provided inFIG. 6(b), image capture element652may capture multiple images, where each image captures at least a portion of the cellular phone610, the reader606, the laptop608and the PDA612. Upon detecting at least a portion of a device based on the captured image(s), the client device650can determine an approximate current location of the detected device, as previously discussed above. The device icons618,620,622and624are displayed on client device650representing the cellular phone602, reader606, laptop608and PDA612. Each of the icons is displayed in the locations of each detected relative to the client device650.

In the instance where the image capture element652also includes a communication element (e.g., IR transceiver), the client device650also can provide a “communication” mode where the image capture element652can be focused to communicate with a particular device. For example, the client device650could adjust the image capture element652such that only a single device, i.e., the cellular phone610, is located within the field of view654. As discussed above, such an approach enables communication without identifier codes or when used with identifier codes, can prevent other devices from receiving and decoding the transmissions.

FIG. 7(a) illustrates an example process for detecting at least one device nearby a client device and communicating with the detected device(s). At box702, a client device begins monitoring the surrounding area. The monitoring can begin automatically or upon request of the user or another appropriate source. As discussed above, a client device monitors the surrounding area, in some embodiments, by capturing one or more images with the device's image capture elements.

At box704, the client device, using the captured image(s), can detect the presence of at least a portion of a device (e.g., a computing device or communication device). Detecting at least a portion of a device in a captured image can involve any process discussed herein, such as image recognition based on ambient light or infrared radiation. In some embodiments, box704may be performed by the client device. In other embodiments, box704is performed by a remote service that receives the captured image(s) from the client device for detection analysis.

When at least a portion of a device is present in a captured image, the client device can determine the location of the detected device relative to the client device, at box706, such as by using image analysis and/or triangulation to determine a direction and relative distance to each device. In some embodiments, box706is performed by a remote service that receives the captured image(s) from the client device for location analysis. At box708, the client device determines an ability to communicate with one or more of the detected devices.

At box712, the client device determines identity information associated with one or more of the detected devices. The client device, in some embodiments, emits a request for identification to a detected device. In this case, the client device can receive a response to the request for identification from the detected device. Alternatively, the detected device might emit a heartbeat or other signal that may be received by the client device, such that the client device does not need to emit a request for identification. Identity information may include information associated with a user of the device (e.g., user's name) or the device itself (e.g., cellular phone, PDA). As discussed above, the client device may obtain additional identity information associated with a user of the device or the device itself by using the initial identity data received from the detected device. Using the identity information (box712) and location information (box706), the client device can display a device icon or user icon on the display representing the detected device, at box714. As discussed above, each icon is visible on the display in a location relative to the client device. The client device may also display identity information for the detected device (when available).

It is not likely that detected devices will remain in the same position relative to the client device as was initially displayed by the client device. If, for example,FIG. 3represented client device310in the same room as the other devices302-308, users of the devices302-308are likely to move about, leave the room, return to a different location in the room and the like. Therefore, the client device can periodically (or continuously) update the location of each detected device visible on display328by updating the displayed location of each user icon or device icon to accurately represent the current location of the detected device relative to the client device. In addition, a new user icon or device icon may be added to the display of the client device when a new device is detected. Similarly, a device may be removed from the display when the device is no longer detected by the client device or is removed by the user of the client device.

At box716, the client device periodically monitors the area surrounding the client device and updates any location and/or identity information associated with each detected device. In general, box716is accomplished by repeating boxes702-714. Using the client device310shown inFIG. 3, for example, the image capture elements of the client device310may periodically (or continuously) capture images. When at least a portion of another device is located in a captured image, the client device (or remote service) determines the location of the detected device (box706).

FIG. 7(b) illustrates that a client device can exchange information with another device it has detected without requiring the user to have any knowledge about the detected device. This is in contrast to several methods of communication known in the art today that require a user to have some knowledge of the other device or the other user in order to communicate with the other user. For example, a user cannot send a text message to another device without knowing the phone number or contact information associated with the other device. As another example, a user cannot send an email to another user without knowing the other user's email address. Here, in contrast, the client device displays icons representing each detected device that a user may select to initiate communication with a particular device without having to know any information about the detected device.

At box750, the client device receives a selection of one of the displayed user icons or device icons (e.g., device icon318inFIG. 3). At box752, upon receiving the selection of the icon, the client device identifies the device associated with the selected icon (e.g., reader302). At box754, the client device establishes communication with the reader302. At box756, the client device is enabled to communicate and exchange information with the reader302. In some embodiments, a user may select only an icon that is available (e.g., icon is displayed in solid lines, a certain color, indicates “available”) to initiate communication with the particular device. In other embodiments, a user may select an icon indicating that a device is unavailable (e.g., hidden lines, a certain color, indicates “unavailable”, etc.) to initiate communication with the particular device. In this case, the client device will attempt to determine if it can communicate with the selected device. If the client device determines that it has an ability to communicate with the selected device, the client device will perform the actions shown in the boxes750-756inFIG. 7(b).

In addition to some of the functionality discussed above, additional uses and features of the client device may be advantageous for a variety of different users.FIG. 8illustrates a client device800that includes five image capture elements820,822,824,826,828and a display screen802.FIG. 8illustrates an example use case whereby a teacher may operate the client device800in a classroom environment of students taking a specific test.

In this example, the teacher, before the test is administered to the students, may use the client device800to detect each student's computing device and/or a computing device corresponding to each desk location in the classroom. Using theFIG. 7(a) example, the image capture elements820-828may capture one or more images prior to administering the test. When at least a portion of a user and/or a computing device is located in a captured image, the client device800(or remote service) determines the location of the detected device, an ability to communicate with the detected device and identity information associated with the detected device. The identity information may allow the teacher to verify that correct students are attending class for the test.

FIG. 8illustrates that an icon804can be displayed on a display screen802of the client device800representing each detected device. Each icon804may be displayed by the client device in the actual location of the student relative to the teacher or within the classroom. Identity information, such as the name of the student associated with the device, is displayed along with the icon804(e.g., Annie G., Jennifer E.). The client device800may transmit the test data to each of the devices using any communication protocol known in the art today. The client device800may also obtain status information from each of the student's devices. For example, a student's device that has not yet received the test may be represented by a certain icon or icon state808.

When the teacher is ready to begin the test, the teacher can issue a command via the client device800that can cause the test or defined version of the test to be distributed/sent to the students and to begin the test on each device. The student's devices can periodically deliver test status information back to the client device800such that the test status is displayed. In one example, as shown inFIG. 8, the client device is reporting that several of the students are “in progress” of taking the test while some of the students have already “completed” the test. In addition to this type of status information displayed along with an icon, an icon may be visibly distinct based on the test status (e.g., a completed icon is visibly different than a waiting icon, etc.). When a student completes the test, a signal can be transmitted from the student's device to the client device800, and a new icon or icon state806can be displayed showing that the student has completed the test. When all the students have completed the test, the teacher can collect all the test data (if not already collected) and close out the test. The teacher can also send a message to one or more selected students.

The client device800can also help the teacher with various other tasks. For example, a teacher can take attendance of her students using the client device. If a student is absent, the student's device will not be detected by the client device, and an icon810can be displayed on screen802indicating to the teacher that the student is missing from class. The client device800(or remote service) may also identify the information that was covered in class that the absent student missed. Then, at a later time, any information that the student missed can be transmitted to that student's device by the client device (or a remote service).FIG. 8illustrates that the client device800may also display an icon812representing the current location of the client device itself (e.g., the teacher is walking up and down the rows of desks to monitor the students).

The ability for a user of the client device to move around while still monitoring other devices can be provided in a number of different ways. In a first approach, the captured image information for each of the devices can be analyzed to determine the relative change in position. In many cases, however, it can be difficult to determine whether the change in location is due to movement of the client device, movement of the detected device(s) or a combination of both. In some embodiments, the client device can include image analysis software that monitors/tracks differences in background portions of the captured image to determine movement of the client device. As mentioned with respect toFIGS. 1-2, certain client devices can include orientation or movement-determining elements such as accelerometers or gyro elements. These elements can detect motion of the client device, and can help to determine whether the relative movement of a detected device is due to movement of the detected device or movement of the client device. For example, inFIG. 8where the user devices will remain primarily stationary at the student's desk location, the client device can determine that the changes in relative position are due to movement of the client device and can update the displayed locations accordingly.

It can be desirable to update the locations of nearby devices relative to the client device.FIG. 9(a) illustrates an example arrangement900wherein three different user devices904(associated with Person X),906(associated with Person Y),908(associated with Person Z) are located nearby the client device902. The location of each device is presented by the client device902relative to the location of the client device itself (shown as icon916). Using the arrangement900shown inFIG. 9(a), icons910,912,914are displayed by the client device902. Each icon910,912,914represents the location of the user device906,904,908, respectively, relative to the client device902.

The client device902can detect and display the motion of another device. For example, as shown inFIG. 9(b), the device904associated with Person X moves to a second location (as shown by the arrow inFIG. 9(b)). Tracking the position of the device904relative to the client device can be performed using any of a number of approaches discussed herein, such as to update the position based on the relative location imaged in subsequent images captured using at least one image capture element of the client device. In some cases, there can be an identifier (e.g., a bar code or symbol) on each device that assists with tracking the particular device in subsequent images captured by the client device, while in other embodiments, image recognition is performed to identify specific devices or users. In some embodiments, each device might send a periodic infrared signal, such as a heartbeat message, that indicates to the client device (and any other devices listening for those heartbeats) the current position of a specific device. Each heartbeat message can include any appropriate information, such as an identifier originally sent to the client device. Various other approaches can be used as well as discussed elsewhere herein. The detected movement of the device904can be displayed by the client device902by updating the location of the icon912representing the location of the device904relative to the client device902. The motion of the icon912may be periodically updated in the display or shown in real-time.

The ability to track relative movement between devices can be used as input for a number of different applications other than those previously discussed. For example,FIGS. 10(a)-10(b) illustrate an example where the relative positions of the devices1002and1004can be used as input for a position-dependent aspect of an application, for example a game or similar application. In this example, a gaming application is executed on both client device1002and competitor device1004. As discussed above, the devices can identify each other using infrared or a similar technology and track the location of the other device. Other communication approaches (e.g., Wi-Fi or Bluetooth) can be used for other aspects of the application as discussed elsewhere herein.

FIG. 10(a) shows a player icon1006(e.g., paddle) displayed on the screen of the client device1002in substantially the location that the competitor device1004is relative to the client device1002. InFIG. 10(a), the player using the client device1002(represented by the paddle in the foreground of the display) has hit the ball over the net. In order for the competing player (using the device1004) to hit the ball back over the net, the competing player must move the device1004until the paddle icon1006is in a position to hit the ball back over the net.

As illustrated by the arrow1010inFIG. 10(b), the competing player, in this instance, would have to move the device1002to the right (from the perspective of the player holding device1002) to position the paddle1006in a position to hit the ball back over the net. Using this example of playing a game, the competing user can move the device1004in the appropriate direction until the location of the paddle1006is substantially in the position shown inFIG. 10(b) and then provide another input to hit the ball, such as press a button, interact with a touch screen or swing the competitor device1004in a motion representing hitting a ball.

FIG. 11illustrates another use for displaying the location of another device relative to the client device.FIG. 11illustrates that the location of each player in a hockey game is represented by the location of the user's device relative to another player's device. In this instance, three users are playing a multi-user hockey game whereby each user is using their own device to play the game. A first user is playing the game using device1102, a second user is playing the game using device1104and a third user is playing the game using device1106. Using the example shown inFIG. 11, the first user is able to view the location of the second player using device1104(player icon1110) and the third player using device1106(player icon1112) relative to device1102(player icon1108).

In the instance of playing a multi-player game (e.g., ping-pong, hockey), it may be useful to define the area or boundaries of the virtual playing field prior to the start of the game, such as the size of the ping-pong table or the size of the ice rink. Defining the boundaries of the playing area will help the players understand the potential area that might have to be traversed during the course of the game. For example, the two players participating in a game of virtual ping-pong may decide to create a virtual ping-pong table that is the size of a tennis court as opposed to the size of a conventional ping-pong table. There are several ways to predefine or determine the size of a playing area including, but not limited to, a global positioning system (GPS), a laser range finder, ultrasound and manually defining a playing boundary on the display of any device participating in the game.

An example of defining the boundaries of a virtual ping-pong table will be used to describe how the boundaries of the table could be defined, according to various embodiments. If the client device included a GPS, a user could walk to each corner of the imaginary ping-pong table and, in some embodiments, indicate each corner of the imaginary table in a “set boundary” mode of the client device. After defining each corner of the ping-pong table, the client device could transmit the coordinates of the boundaries of the virtual ping-pong table to the other player's device. In other embodiments, when the boundaries of a game playing field are limited by the size of a room or area, a user, using a GPS, could walk to each corner of the room and designate each corner's coordinates as a boundary. In yet other embodiments, a laser range finder (that is either part of the client device or a separate device) could determine the boundaries of a room and transmit the boundaries to each device participating in the game. In some embodiments, a client device could access the floor plan of the building the players are located within and allow the user to designate what portion of the floor plan (including the entire room) constitutes the boundaries for the game (e.g., size of the ping-pong table).

In the instance where the players are not located in the same room, each device could display the floor plan of the particular room that each player was going to play the game within, and that user could draw or designate what portion of the room would be the boundary. This would allow users to create larger or smaller boundaries for themselves. For example, a first user may want more exercise/challenge when playing virtual ping-pong and therefore designates a larger playing area requiring the player to cover more territory during the course of the game (e.g., sets width of his half of the ping-pong table as 10 ft wide). On the other hand, the second player in the game might designate a smaller width for her half of the virtual ping-pong table. Thus, the second player could more easily cover their half of the table. This may be especially useful when the two players are of different athletic abilities and/or age. For example, a father that wants to play virtual ping-pong with his ten year-old son may set the width of his half of the ping-pong table to twice the width of his son's half of the ping-pong table so that the game is easier for his son and more challenging for himself. Different size playing fields could also be accomplished using a GPS, laser range finder or ultrasound.

Using the example of the father and son playing a virtual game of ping-pong, suppose the father and son want to play the game in the backyard. In some embodiments, the father's device may automatically identify each of the game participants (himself and his son), log each participant into the game and configure the game. Once the game is configured, the father's device may transmit the game configuration information to his son's device. In other embodiments, each of the participants' devices (father and son's device) identifies the other person as a game participant, logs themselves into the game and configures the game. For example, while in a game “configuration” mode, in one embodiment, the father's device may detect that his son is present in the backyard, configure the game for a two-player mode, log each of them into the game and automatically define the boundaries of the virtual ping-pong table within the backyard. The game configuration information can then be transmitted to the son's device. The father's device may personalize the game experience for the father by, for example, identifying an avatar or icon associated with the father (and using that avatar/icon to represent him in the game), the father's high score (to infer skill level), his preference to use a left-handed control, etc. The son's device may personalize the game in a similar manner for the son. These personal attributes identified by a device are exemplary only and are not intended to limit the scope of the technology described herein.

Returning toFIG. 4(c), the user of the client device can select one or more icons representing detected devices to receive information (e.g., a file) simply by selecting the appropriate icons displayed on the display of the client device.FIG. 12illustrates an example information page1200that might be displayed on a client device in response to the selection of icons466and468shown inFIG. 4(c). The selected icons are displayed in the information page (e.g., user icon1202and device icon1206) as well as the identity information associated with the icon (e.g., user information1204and device information1208). The information display in the information page1200can help the user operating the client device ensure that information (e.g., Presentation.ppt) will be transmitted to the appropriate devices. The information page1200can also include any other appropriate features, such as a browse button1210used to select the file that will be sent to those selected devices, provide comments and/or instructions in a comment box1212or any other appropriate information. The file or information can be sent to another device using any appropriate communication technology and need not be transmitted using the approach that was used to determine the identity and/or location of the nearby devices.

In some embodiments, each item of identity information associated with a device and/or user may be selectable. This feature may be especially useful if, for example, a user would like to send information to all known persons associated with a particular group, company, etc.FIG. 13illustrates an example of an information page1300that may be displayed upon selection of “Products Intl.” in the identity information467displayed along with icon466shown inFIG. 4(c). Information page1300displays a group of persons1302associated with Products Intl.: Mary Lee, Catherine Jones, Steve Adams, Bob Wilson, Mike Parker and Tina Westwood. The group1302may consist of all persons in, by way of example only, the user's address book, contacts, social network, etc., all persons associated with a currently detected device or a combination thereof, that are associated with the company Products Intl. In the example shown inFIG. 13, a selection box1304is displayed along with each person in the group1302. In some embodiments, the selection box1304for person displayed in the group1302is selected by default. In this instance, the user may browse available files using the browse button1310, select the desired file (e.g., Presentation.ppt) and select the “send” button to send the “Presentation.ppt” file to each person in the group1302. The user may also add comments in comments window1312that would be sent along with the selected file. In other embodiments, the selection box1304for each person displayed in the group1302is not automatically selected by default and the user must select the box1304for each person individually in order to send the “Presentation.ppt” file to the person.

FIG. 14illustrates an example interface page1400that enables a user of a client device to specify various aspects of the detection and communication processes of the client device. It should be understood thatFIG. 14merely includes a sample selection of configurable options1402for explanation purposes, and that a number of additional or alternative parameters or options could be utilized within the scope of the various embodiments. In this example, the user is able to activate or deactivate the automatic detection of nearby devices, as well as specify ways in which the device detects and/or connects with these nearby devices. In this example, the user has specified to have the client device always scanning for client devices using at least one infrared element of the client device. The user has also indicated that the client device should connect to these detected devices only when instructed by the user and that once detected and identified, the devices should communicate using Bluetooth® protocol. As should be understood, various options for each element can be provided as discussed or suggested elsewhere herein.

In accordance with the various embodiments discussed above, various environments and technologies can be utilized as appropriate or desirable for various applications. For example,FIG. 15illustrates an example of an environment1500for implementing aspects in accordance with various embodiments. As will be appreciated, although a Web-based environment is used for purposes of explanation, different environments may be used, as appropriate, to implement various embodiments. The environment1500shown includes a variety of client devices1502, which can include any appropriate device operable to send and receive requests, messages or information over an appropriate network1504and convey information back to a user of the device. As discussed, at least some of these devices can also include components enabling the devices to communicate directly with each other via wireless communication. Examples of such client devices include personal computers, cell phones, handheld messaging devices, laptop computers, set-top boxes, personal data assistants, electronic book readers and the like. Each client device can be capable of running at least one motion or orientation-controlled interface as discussed or suggested herein. In some cases, at least some functionality will be generated and/or managed on at least one client device. In other embodiments, at least some of the functionality or content will be generated and/or managed over at least one network1504.

The network1504can include any appropriate network, including an intranet, the Internet, a cellular network, a local area network, or any other such network or combination thereof. Components used for such a system can depend at least in part upon the type of network and/or environment selected. Protocols and components for communicating via such a network are well known and will not be discussed herein in detail. Communication over the network can be enabled by wired or wireless connections and combinations thereof. In this example, the network includes the Internet, as the environment includes a content provider system including at least one Web server1506for receiving requests from a client device1502and serving content in response thereto, although for other networks an alternative device serving a similar purpose could be used as would be apparent to one of ordinary skill in the art.

Each data store can include several separate data tables, databases, or other data storage mechanisms and media for storing data relating to a particular aspect. For example, a content data store1512as illustrated includes content that might be accessed or shared by a user of a client device. The user data store1516includes identification data and/or security information useful for validating specific users or associating users with specific devices. A session data store1514can store any appropriate information for a session, such as which devices participated or were detected, what information was obtained for each device, etc. It should be understood that there can be many other aspects that may need to be stored in a data store, such as access rights information, which can be stored in any of the above listed mechanisms as appropriate or in additional mechanisms in the data store. Each data store is operable, through logic associated therewith, to receive instructions from a respective application server and obtain, update or otherwise process data in response thereto. In one example, a user might submit a search request for a certain type of content. In this case, the data store might access the user information to verify the identity of the user and can access the content information to obtain information about instances of that type of content. The information can then be returned to the user, such as in a message or on a Web page that the user is able to view via a browser on the client device1502.

Each server will typically include an operating system that provides executable program instructions for the general administration and operation of that server and will typically include a computer-readable medium storing instructions that, when executed by a processor of the server, allow the server to perform its intended functions. Suitable implementations for the operating system and general functionality of the servers are known or commercially available and are readily implemented by persons having ordinary skill in the art, particularly in light of the disclosure herein.

The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereunto without departing from the broader spirit and scope of the invention as set forth in the claims. For example, even though the client device was described throughout as a portable electronic device, the technology described herein may also apply to a non-portable electronic device (e.g., desktop computer, game console).