Patent ID: 12226696

DETAILED DESCRIPTION

The present invention relates to an enhanced gaming experience. Current gaming systems, such as POKÉMON GO, utilize a phone, a camera, and an image inserted in front of an object. In operation a mobile device such as a phone is used which includes a camera. A user may observe video acquired by the phone by viewing the display of the phone. As a part of the gaming experience, an image may be inserted by the game in front of a video representation of an actual object on the display of the mobile device. Thus, a user may, by viewing the display of the mobile device, see actual video imagery acquired with the camera as well as additional imagery added as a part of the game. Game imagery may be added based on the location of the user such as through using GPS location data. Thus, when a user is at a specific location they may be able to view specific game imagery which is mapped to that location.

According to one aspect, one or more ear pieces or headsets are used to enhance audio aspects of the gaming experience. The one or more earpieces or headsets may use an AUDIO TRANSPARENCY feature where ambient audio is sensed at one or more microphones of the earpiece and then reproduced at one or more speakers of the earpiece. Thus, even though a user may be wearing the earpiece or headset they are still able to hear ambient or environmental sound. In addition, game sounds may be added to the ambient or environmental sounds. The game sounds may be mapped to specific locations. Thus, when the user is at a specific location, they may hear a particular sound. In addition, sound processing may be performed based on the relative location of the user and the location of the source of the sound. Thus, for example, the sound may be heard when the user is at a set distance from the location of the source of the sound and the intensity of the sound may increase as the user approaches the location of the source of the sound. Similarly, 3D sound processing may be performed based on the relative location of the user (and both the right and left ears of the user) to the location of the source of the sound thus, for example, a user may hear the sound in a manner which provides them with spatial context as to where the source of the sound is located relative to their current position.

FIG.1displays a user106experiencing enhanced gaming with the audio transparency feature. The object image20on the phone104represents a clue or object in the game system94. The phone104is not limited to a phone or smart phone, but may encompass other electronic devices, including but not limited to other mobile devices, iPADs, tablets, game consoles, etc.FIG.1displays two electronic devices intended to show the detail of two systems or two aspects of a system operating simultaneously within one device. The game system94may encompass all gaming applications or systems played on or incorporating any electronic devices. As the user106moves, the object image20on the phone104may move in response to updated position information received via a geospatial navigation system such as a global positioning (“GPS”) system receiver or other location tracking system. Of course, other types of navigation systems may be used. For example, a GLOSNASS position may be provided, a position may be interpolated based on nearby cell phone towers, a position may be determined based on one or more WiFi routers within range, a position may be determined based on a combination of one or more cell phone towers within range and one or more WiFi routers within range, the position may be determined based on sensing movement via one or more inertial sensors, or the position may be otherwise determined. The GPS signal96received at a GPS receiver of the phone104may be used to track the movement of the user106. The game system94coordinates the user's movement as determined using the GPS signal96with the object image20on the phone. As the user106approaches a certain point, shown inFIG.1as the user location100, the phone may vibrate or make a sound to alert the user106. This sound is depicted inFIG.1as the game sound110. Upon hearing the game sound110, the user106has options, one of which is to continue the game by touching the screen of the electronic device104to activate the object image20or other generic images112in the game system94. The user106may also approach a certain point and hear only the environmental sounds, as transmitted clearly from the wireless earpiece speakers to the internal microphone. Or, the user may reach a certain location and hear the game sounds overlapping with the environmental sounds. Actions of the user may vary depending on the particular gaming experience. In one embodiment of the enhanced gaming experience, a new sound would be inserted in front of the game sound110. This new sound, depicted inFIG.1as environmental sound108, would be framed by the position of the phone104and its sensors102. This environmental sound108may come from the surrounding environment of the user106, from the outside world, or it may be a sound generated by another electronic device such as a phone, an earpiece, or another external device. In one embodiment, the microphone of the earpiece(s)120senses the environmental sound108or the external sound.

FIG.2displays an overview of the audio transparency technology.FIG.2depicts the convergence of the environmental sound108with the game sound110to form the composite audio105. The wireless earpiece(s)120, described in further detail later herein sense the environmental sound108. The environmental sound108and the game sound110are combined, and the user106experiences the composite audio105. This composite audio may be considered a feature of AUDIO TRANSPARENCY. The overlap of sounds extends to sounds from the game combined with sounds from other sources, such as other external devices or other users. Other earpiece users may participate in the game with the user, and the audio messages these users send may overlap with the game sounds or the environmental sounds in an ambient audio format. External speakers may transmit environmental sounds to the microphone of the user's earpieces. At the same time, the user may hear audio messages sent from other earpiece users. These audio messages may be restricted or require acceptance from the user before the message is relayed, depending on the user's earpiece settings.

FIG.3is a flowchart depicting one use of the earpiece microphones to create the composite audio105. InFIG.3, the external microphones122of the earpiece(s)120may convey the environmental sounds108to the processor(s)128, internal microphones124, speaker126, or the wireless transceivers130. As the external microphone122conveys the environmental sound108, the internal or bone microphone124may convey the game sound110. As part of the audio transparency technology, the user106may experience game sounds110overlapped with environmental sounds108. The composite audio may also be created as the wireless transceiver relays audio messages from other earpiece users, and the microphone conveys environmental sounds. Or the internal microphone of the wireless earpiece may convey sounds from other sources, such as music saved on an electronic device, or music streamed from a server, as the wireless transceiver conveys audio messages. The composite audio effect may also be created by layering the game sounds with music from another device and environmental sounds.

FIG.4depicts one non-limiting example of this audio transparency technology.FIG.4shows a user106looking at a house84, a tree80, and several bushes82in front of the house. The audio from the world (in this example, sounds from the house and surrounding foliage), would be transmitted normally to the earpiece or another device as the environmental sound108. As the user discovers the clue or item being searched for, shown inFIG.4as the object image20, the game alerts the user by introducing its own sound, the game sound110, sonically positioned where the item would be able to be detected. No gaming experiences currently overlay game sounds with environmental sounds. One type of enhanced experience is created as game sounds110are interlaced with environmental sounds108transmitted using audio transparency technology which provides for sensing environmental sounds108using one or more earpieces120and then reproducing the environmental sounds108using one or more microphones of the earpiece. Although the example above describes outdoor sounds, environmental sounds may include all varieties of surrounding environments, including but not limited to: musical sounds, nightlife, churches, temples, and cathedrals, bells, airport sounds, fairground sounds, the sounds of sports games, travel sounds, the sounds of storms, environmental sounds from museums and libraries, birdsongs, animal calls, etc. Generic images112may also create game sounds different from those created by the object image20, and these generic sounds may also be interlaced with the game sound110and the environmental sounds108to create another form of the composite audio sound105. Generic images in the game may correspond with the environmental sounds. The images in the game may change according to the environmental sound the user is hearing. For example, if a user is near a cathedral, the GPS locator or the camera may capture this image. Then the earpiece may pick up the sound of the bell from the surrounding environment. Or, the earpiece may signal the user's electronic device or send a signal to a server to reproduce the sound of a bell. As the user hears the bell through the earpiece, a signal may be sent from the earpiece to the game to change the image on the phone to a cathedral (the image that corresponds to the sound of the bell).

FIG.5displays the wireless earpieces120, also labeled earpieces120.FIG.5illustrates the earpiece sensors, which may include several different forms of sensors, to be used together or individually, including but not limited to inertial sensors, accelerometers, gyroscopes, photoelectric sensors, etc. The sensors may transmit different information to the earpieces, including but not limited to software instructions, data transmission, wireless intrabody communication, biomedical monitoring, etc. The sensors may track the heart rate, steps, or other bodily movements and activity of the user throughout the game. The sensors may be configured to work with the earpieces according to certain user preferences. For example, the sensors may be preset to work only when the user is playing the game. They may be preset to work only via galvanic communication, or only on battery power. The earpieces may transmit data from the sensors to the user via audio messages, or by sending messages to the user's electronic device. The earpieces may also transmit data from the sensors to smart glasses, where the user may read the data across the screen of the glasses. The earpieces may communicate with the sensor in different ways, depending on the configuration of the sensor.FIG.6displays a user106wearing the earpieces, and the earpieces communicating with a phone104and computer. Other electronic devices, though not included inFIG.6, may communicate with the earpieces120.

FIG.7shows another embodiment of an enhanced gaming experience, as the game sound110is positioned within a three-dimensional audio environment.FIG.7depicts the simulation of a three-dimensional audio environment. The earpieces120, worn by the user106(not pictured), are surrounded by multiple sound sources. The embodiment is not limited to the three sources displayed inFIG.7. The three sources occupy three different planes in space, displayed in the figure as the X, Y, and Z PLANES. The game sound110in the Y-plane appears to the user106to come from the game sound source76. The environmental sound source74gives the illusion of the environmental sound108emanating from the Z-plane. And an alternate audio sound107appears to come from the alternate sound source72in the X-plane. By performing sound localization within the x, x-y or alternately the x-y-z planes, the user is provided with a more immersive experience. The audio signal is transduced to provide a three-dimensional audio experience for the user. The audio sounds appear to come from above, behind, and below the user. The effect of tricking the user's brain or simulating actual three-dimensional (“3D”) audio may be created with game sounds coming from the phone or another electronic device (iPAD, tablet, computer, etc.), in combination with sounds coming from the user's wearable earpieces, and from the actual outside environment, enhanced with the wireless earpieces. The effect may occur via speakers inside the earpieces which use head-related transfer function, HRTF filters, cross talk cancellation techniques, and other devices that mimic the appearance of sound waves emanating from a point in the user's three-dimensional space.FIG.3depicts one mechanism by which the wireless earpieces may be used to create this three-dimensional audio effect. The external microphone122brings in the noise of the outside world or the user's surroundings, creating an environmental sound108. At the same time, the internal or bone microphones124may convey the game sound110and an alternate audio sound107. A three-dimensional audio transparency effect is created as the game sound110is overlaid with the environmental sounds110and the alternate audio sound107. If environmental sounds108are combined with game sounds110, a two-dimensional audio effect can be created as well.

In another embodiment, shown inFIG.8, the audio and imaging data streams are separated before they reach the user106. This separation of audio and visual streams creates an augmented reality. The camera98of the phone104would be detached so that it could not immediately transfer information to the phone in conjunction with the audio system. For example, if the camera98was pointed toward the ground and the user was expecting a certain sound to emanate, the separation of audio and visual streams would allow the user to turn his/her/its head and pick up audio that would not necessarily be the same as that which would be detected by the camera-based phone position. The earpieces120may contain inertial sensors404to detect the user's head movements and generate a signal to emanate a certain sound. With the user's fixed point of reference created by disconnecting the camera, the earpieces allow the user, with a turn of the head (user head movement305), to detect different audio streams from the point of view of the inertial sensors in the earpiece. The earpiece settings allow the user106to distinguish the camera-based outputs from the augmented audio reality which incorporates the audio transparency and sound features. This separation of audio and visual reality may also be created via smart glasses sharing data with the earpieces. The smart glasses may change the user's view of the game, or incorporate environmental images into the screen of the glasses. Overlapping the visual images in the game with images portrayed in the smart glasses may correspond with the overlap of audio effects between the earpieces, and the environment and game sounds. This overlap of visual and audio images created by the smart glasses and the wireless earpieces may create a different but enhanced gaming experience for the user.

Another embodiment of the enhanced gaming experience, depicted inFIG.9, utilizes inertial sensors which measure and report the movement of the user. Such sensors may indicate measurements including but not limited to the user's specific force, angular rate, magnetic field, etc. The inertial sensor(s)404may be located on the earpiece, or another wearable device of the user. As the inertial sensor404detects the user movement, the sensor may relay a signal to the phone to generate a game sound110based on the user movement. The inertial sensor404may also relay a signal303to the microphones300of the earpiece120or other wearable device to generate a sound. One or multiple inertial sensors404may be used to track the head position of a user (depicted inFIG.10as user head movement305) and to generate a game or environmental sound based on the head position of the user. Examples of inertial sensors404include but are not limited to accelerometers, gyroscopes, magnetometers, etc. The inertial sensors may be used with the earpieces and the gaming system to create a unique gaming experience for the user. As the user moves his/her head, the inertial sensors may send signals to the wireless earpieces. The earpieces may generate sounds when the user's head is not turned toward the game. Or the earpieces may generate sounds as the user moves and controls the image within the game.

Therefore, various methods, systems, and apparatus have been shown and described. Although specific examples and embodiments are set forth herein it is to be understood that the present invention contemplates numerous variations, options, and alternatives.