Interaction of user devices and servers in an environment

In an environment where video display devices are available for simultaneous viewing, servers provide audio streams to user devices for listening. People may, thus, listen to the audio streams through their user devices while watching the video display devices. An application on the user devices determines which audio streams are available, and the servers may indicate which audio streams are available. The application sends a request to a server to transmit a selected audio stream. The server transmits the selected audio stream, e.g. over a wireless network in the environment. A variety of additional features are enabled by the interaction of the application and the servers.

BACKGROUND

A television generally provides both video and audio to viewers. In some situations, such as in a gym, restaurant/bar, airport waiting area, etc., multiple TVs or other video display devices (each with different video content) may be provided for public viewing to multiple clients/patrons in a single large room. If the audio signals of each TV were also provided for public listening in these situations, the noise level in the room would be intolerable and the people would not be able to distinguish the audio from any single TV nor the voices in their own personal conversations. Consequently, it is preferable to mute the audio signals on each of the TVs in these situations in order to prevent audio chaos. Some of the people, however, may be interested in hearing the audio in addition to seeing the video of some of the display devices in the room, and each such person may be interested in the program that's on a different one of the display devices.

One suggested solution is for the close captioning feature to be turned on for some or all of the display devices, so the people can read the text version of the audio for the program that interests them. However, the close captions are not always a sufficient solution for all of the people in the room.

Another suggested solution is for the audio streams to be provided through relatively short-distance or low-power radio broadcasts within the establishment wherein the display devices are viewable. Each display device is associated with a different radio frequency. Thus, the people can view a selected display device while listening to the corresponding audio stream by tuning their radios to the proper frequency. Each person uses headphones or earbuds or the like for private listening. For this solution to work, each person either brings their own radio or borrows/rents one from the establishment.

In another solution in an airplane environment, passengers are provided with video content on display devices while the associated audio is provided through a network. The network feeds the audio stream to an in-seat console such that when a user plugs a headset into the console, the audio stream is provided for the user's enjoyment.

SUMMARY

In an environment where video display devices are available for simultaneous viewing by multiple people, servers provide audio streams to user devices for individual private listening. People may, thus, listen to the audio streams through their user devices while watching the video display devices. An application on the user devices determines which audio streams are available, and the servers may indicate which audio streams are available. The application sends a request to a server to transmit a selected audio stream. The server transmits the selected audio stream, e.g. over a wireless network in the environment. A variety of additional features are enabled by the interaction of the application and the servers.

DETAILED DESCRIPTION

In some embodiments, the solution described herein allows a user to utilize a personal portable device such as a smartphone to enjoy audio associated with a public display of video. The portable device utilizes a software application to provide the association of audio with the public video. Therefore, the present solution does not require very specific hardware within the seats or chairs or treadmills or nearby display devices, so it is readily adaptable for a restaurant/bar or other establishments.

An environment100incorporating a variety of audio-video (NV) equipment in accordance with an embodiment of the present invention is shown inFIG. 1. In general, the environment100includes one or more video display devices101available for viewing by multiple people/users102, at least some of whom have any one of a variety of user devices that have a display (the user devices)103. Video streams (at least one per display device101), such as television programs, Internet-based content, VCR/DVD/Blue-ray/DVR videos, etc., are generally provided through the display devices101. The users102may thus watch as many of the video streams as are within viewing range or as are desired. Additionally, multiple audio streams corresponding to the video streams (generally at least one for each different video stream) are made available through a network (generally including one or more servers104and one or more network access points105) accessible by the user devices103. The users102who choose to do so, therefore, may select any available audio stream for listening with their user devices103while watching the corresponding video stream on the corresponding display device101.

The environment100may be any place where video content may be viewed. For example, in some embodiments, the environment100may be a public establishment, such as a bar/pub, restaurant, airport lounge/waiting area, medical waiting area, exercise gym, outdoor venue, concert arena, drive-in movie theater or other establishment that provides at least one display device101for customer or public viewing. Users102with user devices103within the establishment may listen to the audio stream associated with the display device101of their choice without disturbing any other people in the same establishment. Additionally, picture-in-a-picture situations may have multiple video streams for only one display device101, but if the audio streams are also available simultaneously, then different users102may listen to different audio streams. Furthermore, various features of the present invention may be used in a movie theater, a video conferencing setting, a distance video-learning environment, a home, an office or other place with at least one display device101where private listening is desired. In some embodiments, the environment100is an unstructured environment, as differentiated from rows of airplane seats or even rows of treadmills, where a user may listen only to the audio that corresponds to a single available display device.

According to some embodiments, the user devices103are multifunctional mobile devices, such as smart phones (e.g. iPhones™, Android™ phones, Windows Phones™, BlackBerry™ phones, Symbian™ phones, etc.), cordless phones, notebook computers, tablet computers, Maemo™ devices, MeeGo™ devices, personal digital assistants (PDAs), iPod Touches™, handheld game devices, audio/MP3 players, etc. Unlike for the prior art solution, described above, of using a radio to listen to the audio associated with display devices, it has become common practice in many places for people to carry one or more of the mobile devices mentioned, but not a radio. Additionally, whereas it may be inconvenient or troublesome to have to borrow or rent a radio from the establishment/environment100, no such inconvenience occurs with respect to the mobile devices mentioned, since users102tend to always carry them anyway. Furthermore, cleanliness and health issues may arise from using borrowed or rented headphones, and cost and convenience issues may arise if the establishment/environment100has to provide new headphones or radio receivers for each customer, but no such problems arise when the users102all have their own user devices103, through which they may listen to the audio. As such, the present invention is ideally suited for use with such mobile devices, since the users102need only download an application (or app) to run on their mobile device in order to access the benefits of the present invention when they enter the environment100and learn of the availability of the application. However, it is understood that the present invention is not necessarily limited only to use with mobile devices. Therefore, other embodiments may use devices that are typically not mobile for the user devices103, such as desktop computers, game consoles, set top boxes, video recorders/players, land line phones, etc. In general, any computerized device capable of loading and/or running an application may potentially be used as the user devices103.

In some embodiments, the users102listen to the selected audio stream via a set of headphones, earbuds, earplugs or other listening device106. The listening device106may include a wired or wireless connection to the user device103. Alternatively, if the user device103has a built-in speaker, then the user102may listen to the selected audio stream through the speaker, e.g. by holding the user device103next to the user's ear or placing the user device103near the user102.

The display devices101may be televisions, computer monitors or other appropriate video or A/V display devices. In some embodiments, the audio stream received by the user devices103may take a path that completely bypasses the display devices101, so it is not necessary for the display devices101to have audio capabilities. However, if the display device101can handle the audio stream, then some embodiments may pass the audio stream to the display device101in addition to the video stream, even if the audio stream is not presented through the display device101, in order to preserve the option of sometimes turning on the audio of the display device101. Additionally, if the display device101is so equipped, some embodiments may use the audio stream from a headphone jack or line out port of the display device101as the source for the audio stream that is transmitted to the user devices103. Furthermore, in some embodiments, some or all of the functions described herein for the servers104and the network access points105may be built in to the display devices101, so that the audio streams received by the user devices103may come directly from the display devices101.

According to some embodiments, each user device103receives a selected one of the audio streams wirelessly. In these cases, therefore, the network access points105are wireless access points (WAPs) that transmit the audio streams wirelessly, such as with Wi-Fi, Bluetooth™, mobile phone, fixed wireless or other appropriate wireless technology. According to other embodiments, however, the network access points105use wired (rather than wireless) connections or a combination of both wired and wireless connections, so a physical cable may connect the network access points105to some or all of the user devices103. The wired connections, however, may be less attractive for environments100in which flexibility and ease of use are generally desirable. For example, in a bar, restaurant, airport waiting area or the like, many of the customers (users102) will likely already have a wireless multifunction mobile device (the user device103) with them and will find it easy and convenient simply to access the audio streams wirelessly. In some embodiments, however, one or more users102may have a user device103placed in a preferred location for watching video content, e.g. next to a bed, sofa or chair in a home or office environment. In such cases, a wired connection between the user device103and the server104may be just as easy or convenient to establish as a wireless connection.

Each server104may be a specially designed electronic device having the functions described herein or a general purpose computer with appropriate peripheral devices and software for performing the functions described herein or other appropriate combination of hardware components and software. As a general purpose computer, the server104may include a motherboard with a microprocessor, a hard drive, memory (storing software and data) and other appropriate subcomponents and/or slots for attaching daughter cards for performing the functions described herein. Additionally, each server104may be a single unit device, or the functions thereof may be spread across multiple physical units with coordinated activities. In some embodiments, some or all of the functions of the servers104may be performed across the Internet or other network or within a cloud computing system.

Furthermore, according to different embodiments, the servers104may be located within the environment100(as shown inFIG. 1) or off premises (e.g. across the Internet or within a cloud computing system). If within the environment100, then the servers104generally represent one or more hardware units (with or without software) that perform services with the A/V streams that are only within the environment100. If off premises, however, then the servers104may represent a variety of different combinations and numbers of hardware units (with or without software) that may handle more than just the A/V streams that go to only one environment100. In such embodiments, the servers104may service any number of one or more environments100, each with its own appropriate configuration of display devices101and network access points105. Location information from/about the environments100may aid in assuring that the appropriate audio content is available to each environment100, including the correct over-the-air TV broadcasts.

The number of servers104that service any given environment100(either within the environment100or off premises) is generally dependent on a variety of factors including, but not limited to, the number of display devices101within the environment100, the number of audio or A/V streams each server104is capable of handling, the number of network access points105and user devices103each server104is capable of servicing and the number of users102that can fit in the environment100. Additionally, the number of network access points105within any given environment100is generally dependent on a variety of factors including, but not limited to, the number of display devices101within the environment100, the size of the environment100, the number of users102that can fit in the environment100, the range of each network access point105, the bandwidth and/or transmission speed of each network access point105, the degree of audio compression and the presence of any RF obstructions (e.g. walls separating different rooms within the environment100). In some embodiments, there may even be at least one server104and at least one network access point105connected at each display device101.

Each server104generally receives one or more audio streams (and optionally the corresponding one or more video streams) from an audio or A/V source (described below). The servers104also generally receive (among other potential communications) requests from the user devices103to access the audio streams. Therefore, each server104also generally processes (including encoding and packetizing) each of its requested audio streams for transmission through the network access points105to the user devices103that made the access requests. In some embodiments, each server104does not process any of its audio streams that have not been requested by any user device103. Additional functions and configurations of the servers104are described below with respect toFIGS. 19-21.

In some embodiments, each of the display devices101has a number, letter, symbol, code, thumbnail or other display indicator107associated with it. For example, the display indicator107for each display device101may be a sign mounted on or near the display device101. The display indicator107generally uniquely identifies the associated display device101. Additionally, either the servers104or the network access points105(or both) provide to the user devices103identifying information for each available audio stream in a manner that corresponds to the display indicators107, as described below. Therefore, each user102is able to select through the user device103the audio stream that corresponds to the desired display device101.

Particularly for, but not necessarily limited to, embodiments in which the environment100is a public venue or establishment (e.g. bar, pub, restaurant, airport lounge area, museum, medical waiting room, etc.), an information sign108may be provided within the environment100to present information to the users102regarding how to access the audio streams for the display devices101and any other features available through the application that they can run on their user devices103. The information sign108may be prominently displayed within the environment100. Alternatively, an information card with similar information may be placed on each of the tables within the environment100, e.g. for embodiments involving a bar or restaurant.

Two examples of an information sign (or card) that may be used for the information sign108are shown inFIGS. 2 and 3. The words shown on the example information sign/card109inFIG. 2and the example information sign/card110inFIG. 3are given for illustrative purposes only, so it is understood that embodiments of the present invention are not limited to the wordings shown. Any appropriate wording that provides any desired initial information is acceptable. Such information may include, but not be limited to, the availability of any of the functions described herein.

For the example information sign/card109inFIG. 2, a first section111generally informs the users102that they can listen to the audio for any of the display devices101by downloading an application to their smart phone or Wi-Fi enabled user device103. A second example section112generally informs the users102of the operating systems or platforms or types of user devices103that the can use the application, e.g. Apple™ devices (iPhone™, iPad™ and iPod Touch™), Google Android™ devices or Windows Phone™ devices. (Other types of user devices may also be supported in other embodiments.) A third example section113generally provides a URL (uniform resource locator) that the users102may enter into their user devices103to download the application (or access a website where the application may be found) through a cell phone network or a network/wireless access point, depending on the capabilities of the user devices103. The network access points105and servers104, for example, may serve as a Wi-Fi hotspot through which the user devices103can download the application. A fourth example section114in the example information sign/card109generally provides a QR (Quick Response) Code™ (a type of matrix barcode or two-dimensional code for use with devices that have cameras, such as some types of the user devices103) that can be used for URL redirection to acquire the application or access the website for the application.

The example information sign/card110inFIG. 3generally informs the users102of the application and provides information for additional features available through the application besides audio listening. Such features may be a natural addition to the audio listening application, since once the users102have accessed the servers104, this connection becomes a convenient means through which the users102could further interact with the environment100. For example, in an embodiment in which the environment100is a bar or restaurant, a first section115of the example information sign/card110generally informs the users102that they can order food and drink through an application on their user device103without having to get the attention of a wait staff person. A second section116generally informs the users102how to acquire the application for their user devices103. In the illustrated case, another QR Code is provided for this purpose, but other means for accessing a website or the application may also be provided.

A third section117generally provides a Wi-Fi SSID (Service Set Identifier) and password for the user102to use with the user device103to login to the server104through the network access point105. The login may be done in order to download the application or after downloading the application to access the available services through the application. The application, for example, may recognize a special string of letters and/or numbers within the SSID to identify the network access point105as being a gateway to the relevant servers104and the desired services. (The SSIDs of the network access points105may, thus, be factory set in order to ensure proper interoperability with the applications on the user devices103. Otherwise, instructions for an operator to set up the servers104and the network access points105in an environment100may instruct the operator to use a predetermined character string for at least part of the SSIDs.) In some embodiments, the application may be designed to ignore Wi-Fi hotspots that use SSIDs that do not have the special string of letters and/or numbers. In the illustrated case, an example trade name “ExXothermic” (used here and in other Figs.) is used as the special string of letters within the SSID to inform the application (or the user102) that the network access point105with that SSID will lead to the appropriate server104and at least some of the desired services. In other embodiments, the SSIDs do not have any special string of letters or numbers, so the applications on the user devices103may have to query every accessible network access point105or hot spot to determine whether a server104is available. The remaining string “@Joes” is an example of additional optional characters in the SSID that may specifically identify the corresponding network access point105as being within a particular example environment100having an example name “Joe's”.

In an embodiment in which the example information sign/card110is associated with a particular table within the environment100, a fourth section118generally identifies the table, e.g. with a letter, symbol or number (in this example, the number 3). An additional QR Code is also provided, so that properly equipped user devices103can scan the QR Code to identify the table. In this manner, the food and/or beverage order placed by the user102can be associated with the proper table for delivery by a wait staff person.

In addition to the example trade name “ExXothermic”, the example information sign/card110shows an example logo119. With such pieces of information, the users102who have previously tried out the application on their user devices103at any participating environment100can quickly identify the current environment100as one in which they can use the same application.

In some embodiment, the servers104work only with “approved” applications. Such approval requirements may be implemented in a similar manner to that of set-top-boxes which are authorized to decode only certain cable or satellite channels. For instance, the servers104may encrypt the audio streams in a way that can be decrypted only by particular keys that are distributed only to the approved applications. These keys may be updated when new versions or upgrades of the application are downloaded and installed on the user devices103. Alternatively, the application could use other keys to request the servers104to send the keys for decrypting the audio streams.

Similarly, in some embodiments, the applications may work only with “approved” servers104. For example, the application may enable audio streaming only after ascertaining, through an exchange of keys, that the transmitting server104is approved.

The downloading of the application to the user devices103is generally performed according to the conventional functions of the user devices103and does not need to be described here. Once downloaded, the exact series of information or screens presented to the users102through the user devices103may depend on the design choices of the makers of the application. For an embodiment using a smart phone or other multifunctional mobile device for the user device103, an example series of views or simulated screenshots of screens of a user interface for the application is provided inFIGS. 4-18. It is understood, however, that the present invention is not necessarily limited to these particular examples. Instead, these examples are provided for illustrative purposes only, and other embodiments may present any other appropriate information, options or screen views, including, but not limited to, any that may be associated with any of the functions described herein. Additionally, any of the features shown for any of the screens inFIGS. 4-18may be optional where appropriate.

In the illustrated example, an initial welcome screen120, as shown inFIG. 4, is presented on a display of the user devices103to the users102upon launching the application on their user devices103. Additionally, an option is provided to the users102to “sign up” (e.g. a touch screen button121) for the services provided by the application, so the servers104can potentially keep track of the activities and preferences of the users102. If already signed up, the users102may “login” (e.g. a touch screen button122) to the services. Alternatively, the users102may simply “jump in” (e.g. a touch screen button123) to the services anonymously for those users102who prefer not to be tracked by the servers104. Furthermore, an example touch screen section124may lead the users102to further information on how to acquire such services for their own environments100. Other embodiments may present other information or options in an initial welcome screen.

In this example, if the user102chooses to “sign up” (button121,FIG. 4), then the user102is directed to a sign up screen125, as shown inFIG. 5. The user102may then enter pertinent information, such as an email address, a username and a password in appropriate entry boxes, e.g.126,127and128, respectively. The user102may also be allowed to link (e.g. at129) this sign up with an available social networking service, such as Internet-based social networking features of Facebook (as shown), Twitter, Google+ or the like (e.g. for ease of logging in or to allow the application or server104to post messages on the user's behalf within the social networking site). Additionally, the user102may be allowed to choose (e.g. at130) to remain anonymous (e.g. to prevent being tracked by the server104) or to disable social media/networking functions (e.g. to prevent the application or server104from posting messages on the user's behalf to any social networking sites). However, by logging in (not anonymously) when they enter an environment100, the users102may garner “loyalty points” for the time and money they spend within the environments100. The application and/or the servers104may track such time and/or money for each user102who does not login anonymously. Thus, the users102may be rewarded with specials, discounts and/or free items by the owner of the environment100or by the operator of the servers104when they garner a certain number of “loyalty points.”

Furthermore, an optional entry box131may be provided for a new user102to enter identifying information of a preexisting user102who has recommended the application or the environment100to the new user102. In this manner, the new user102may be linked to the preexisting user102, so that the server104or the owners of the environment100may provide bonuses to the preexisting user102for having brought in the new user102. The users102may also garner additional “loyalty points” for bringing in new users102or simply new customers to the environment100. The users102may gain further loyalty points when the new users102return to the environment100in the future.

After entering all of the pertinent information and selecting the various options, the user102may press a touch screen button132to complete the sign up. Alternatively, the user102may prefer to return to the initial welcome screen120by pressing another touch screen button133(e.g. “Home”). Other embodiments may offer other sign up procedures or selections.

In this example, if the user102chooses to “login” (button122,FIG. 4), then the user102is directed to a login screen134, as shown inFIG. 6. The user102thus enters an email address (e.g. at135) and password (e.g. at136) using a touch screen keyboard (e.g. at137). There is also an option (e.g. at138) for the user102to select when the user102has forgotten the password. Furthermore, there is another option for the user102to set (e.g. at139) to always login anonymously or not. There is a touch screen button “Done”140for when the user102has finished entering information or making selections. Additionally, there is a touch screen button “Home”141for the user102to return to the initial welcome screen120if desired. Other embodiments may offer other login procedures or selections.

In this example, after the user102has signed up or logged in, the user device103presents a general action selection screen142, as shown inFIG. 7, wherein the user102is prompted for an action by asking “What would you like to do?” “Back” (at143) and “Cancel” (at144) touch screen buttons are provided for the user102to return to an earlier screen, cancel a command or exit the application if desired. An option to order food and drinks (e.g. touch screen button145) may lead the user102to another screen for that purpose, as described below with respect toFIGS. 14-18. An option (e.g. touch screen button146) may be provided for the user102to try to obtain free promotional items being given away by an owner of the environment100. Touching this button146, thus, may present the user102with another screen (not shown) for such opportunities.

An option (e.g. touch screen button147) to make friends, meet other people and/or potentially join or form a group of people within the environment100may lead the user102to yet another screen (not shown). Since it is fairly well established that customers of a bar or pub, for example, will have more fun if they are interacting with other customers in the establishment, thereby staying to buy more products from the establishment, this option may lead to any number or combinations of opportunities for social interaction by the users102. Any type of environment100may, thus, reward the formation of groups of the users102by providing free snacks, munchies, hors d′oeuvres, appetizers, drinks, paraphernalia, goods, services, coupons, etc. to members of the group. The users102also may come together into groups for reasons other than to receive free stuff, such as to play a game or engage in competitions or just to socialize and get to know each other. The application on the user devices103, thus, may facilitate the games, competitions and socializing by providing a user interface for performing these tasks. Various embodiments, therefore, may provide a variety of different screens (not shown) for establishing and participating in groups or meeting other people or playing games within the environment100. Additionally, such activities may be linked to the users' social networks to enable further opportunities for social interaction. In an embodiment in which the environment100is a workout gym, for example, a user102may use the form-a-group button147to expedite finding a workout partner, e.g. someone who generally shows up around the same time as the user102. A user102could provide a relationship status to other users102within the gym, e.g. “always works alone”, “looking for a partner”, “need a carpool”, etc.

The formation of the groups may be done in many different ways. For example, the application may lead some users102to other users102, or some users102may approach other customers (whether they are other users102or not) within the environment100, or some users102may bring other people into the environment, etc. To establish multiple users102as a group, the users102may exchange some identifying information that they enter into the application on their user devices103, thereby linking their user devices103into a group. In order to prevent unwanted exchange of private information, for example, the server104or the application on the user devices103may randomly generate a code that one user102may give to another user102to form a group. Alternatively, the application of one user device103may present a screen with another QR Code of which another user device103(if so equipped) may take a picture in order to have the application of the other user device103automatically link the user devices103into a group. Other embodiments may use other appropriate ways to form groups or allow users102to meet each other within environments100.

An option to listen to one of the display devices101(e.g. “listen to a TV” touch screen button148) may lead the user102to another screen, such as is described below with reference toFIG. 8. Another option (e.g. touch screen button149) to play a game (e.g. a trivia game, and with or without a group) may lead the user102to one or more additional screens (not shown). Another option (e.g. touch screen button150) to modify certain settings for the application may lead the user102to one or more other screens, such as those described below with reference toFIGS. 11-13. Furthermore, another option (e.g. touch screen button151) to call a taxi may automatically place a call to a taxi service or may lead the user102to another screen (not shown) with further options to select one of multiple known taxi services that operate near the environment100.

Other embodiments may include other options for general functions not shown inFIG. 7. For example, for an embodiment in which the environment100is an exercise gym or facility, the application may provide an option for the user102to keep track of exercises and workouts and time spent in the gym. In another example, for an embodiment in which the environment100is a bar, the application may provide an option for the user102to keep track of the amount of alcohol the user102has consumed over a period of time. The alcohol consumption data may also be provided to the server104in order to alert a manager or wait staff person within the environment100that a particular user102may need a free coffee or taxi ride.

In addition to the other options described herein, a set of icon control buttons152-157that may be used on multiple screens are shown at the bottom of the general action selection screen142. For example, a home icon152may be pressed to take the user102back to an initial home screen, such as the initial welcome screen120or the general action selection screen142. A mode icon153may be pressed to take the user102to a mode selection screen, such as that described below with respect toFIG. 11. A services icon154, similar to the function of the “order food and drinks” touch screen button145described above, may be pressed to take the user102to a food and drink selection screen, as described below with respect toFIGS. 14-18. A social icon155, similar to the “make friends or form a group” touch screen button147described above, may be pressed for a similar function. An equalizer icon156may be pressed to take the user102to an equalizer selection screen, such as that described below with respect toFIG. 12. A settings icon157may be pressed to take the user102to a settings selection screen, such as that described below with respect toFIG. 13. Other embodiments may use different types or numbers (including zero) of icons for different purposes.

Furthermore, the general action selection screen142has a mute icon158. If the application is playing an audio stream associated with one of the display devices101(FIG. 1) while the user102is viewing this screen142, the user102has the option of muting (and un-muting) the audio stream by pressing the mute icon158. In some embodiments in which the user device103is a smart phone, the mute function may be automatic when a call comes in. On the other hand, in an embodiment in which the environment100is a movie theater and the user device103is a smart phone, the application on the user device103may automatically silence the ringer of the user device103.

In this example, after the user102has signed up, logged in or made an appropriate selection (such as pressing the “listen to a TV” touch screen button148, mentioned above), the application on the user device103presents a display device selection screen159, as shown inFIG. 8. This selection screen159prompts the user102to select one of the display devices101for listening to the associated audio stream. Thus, the display device selection screen159presents a set or table of display identifiers160.

The display identifiers160generally correspond to the numbers, letters, symbols, codes, thumbnails or other display indicators107associated with the display devices101, as described above. In the illustrated example, the numbers 1-25 are displayed. The numbers 1-11, 17 and 18 are shown as white numbers on a black background to indicate that the audio streams for the corresponding display devices101are available to the user device103. The numbers 12-16 and 19-25 are shown as black numbers on a cross-hatched background to indicate that either there are no display devices101that correspond to these numbers within the environment100or the network access points105that service these display devices101are out of range of the user device103. The user102may select any of the available audio streams by pressing on the corresponding number. The application then connects to the network access point105that services or hosts the selected audio stream. The number “2” is highlighted to indicate that the user device103is currently accessing the display device101that corresponds to the display indicator107number “2”.

In some embodiments, the servers104may provide audio streams not associated with any of the display devices101. Examples may include Pandora™ or Sirius™ radio. Therefore, additional audio identifiers or descriptors (not shown) may be presented alongside the display identifiers160.

The application on the user device103may receive or gather data that indicates which display identifiers160should be presented as being available in a variety of different ways. For example, the SSIDs for the network access points105may indicate which display devices101each network access point105services. In some embodiments, if the network access points105each service only one display device101, then the display indicator107(e.g. a number or letter) may be part of the SSID and may follow immediately after a specific string of characters. For example, if the application on the user device103receives an SSID of “ExX12” from a network access point105, the application may interpret the string “ExX” as indicating that the network access point105is connected to at least one of the desired servers104and that the audio stream corresponding to the display device101having the display indicator107of number “12” is available. In other embodiments, if the network access points105service more than one display device101, but each display indicator107is guaranteed to be only a single character, then an SSID of “ExX034a” may indicate that the network access point105services the display devices101that have the display indicators107of numbers “0”, “3” and “4” and letter “a”. In another embodiment, if the network access points105service more than one display device101, and each display indicator107is guaranteed to be no bigger than three characters, then an SSID of “ExX005007023” may indicate that the network access point105services the display devices101that have the display indicators107of numbers “5”, “7” and “23”. In another embodiment, an SSID of “ExX#[5:8]” may indicate that the network access point105services the display devices101that have the display indicators107of numbers “5”, “6”, “7” and “8”.

In some embodiments, however, the SSIDs do not indicate which display devices101each network access point105services. In such cases, the application on the user devices103may have to login to each accessible network access point105and query each connected server104for a list of the available display indicators107. Each of the network access points105may potentially have the same recognizable SSID in this case. Other embodiments may user other techniques or any combination of these and other techniques for the applications on the user devices103to determine which display identifiers160are to be presented as available. If the operating system of the user device103does not allow applications to automatically select an SSID to connect to a network access point105, then the application may have to present available SSIDs to the user102for the user102to make the selection.

A set of page indicator circles161are also provided. The number of page indicator circles161corresponds to the number of pages of display identifiers160that are available. In the illustrated example, three page indicator circles161are shown to indicate that there are three pages of display identifiers160available. The first (left-most) page indicator circle161is fully blackened to indicate that the current page of display identifiers160is the first such page. The user102may switch to the other pages by swiping the screen left or right as if leafing through pages of a book. Other embodiments may use other methods of presenting multiple display identifiers160or multiple pages of such display identifiers160.

Additionally, other embodiments may allow other methods of selecting an audio stream. For example, If the user device103contains a camera, the channel selection can be done by a bar code or QR Code on the information sign108(FIG. 1) or with the appropriate pattern recognition software by pointing the camera at the desired display device101or at a thumbnail of the show that is playing on the display devices101. There may also be other designators which may include electromagnetic signatures.

Alternatively, the application may switch to a different audio stream based on whether the user points the camera of the user device103at a particular display device101. Also, low-resolution versions of the available video streams could be transmitted to the user device103, so the application can correlate the images streamed to the user device193and the image seen by the camera of the user device103to choose the best display device101match. Alternatively, the image taken by the camera of the user device103may be transmitted to the server104for the server104to make the match.

In other embodiments, a motion/direction sensor, e.g. connected to the user's listening device, may determine which direction the user102is looking, so that when the user102looks in the direction of a particular display device101, the user102hears the audio stream for that display device101. Additionally or in the alternative, when the user102looks at a person, a microphone turns on, so the user may hear that person. A locking option may allow the user102to prevent the application from changing the audio stream every time the user102looks in a different direction. In some embodiments, the user102may toggle a touch screen button when looking at a particular display device101in order to lock onto that display device101. In some embodiments, the application may respond to keying sequences so that the user102can quickly select a mode in which the user device103relays an audio stream. For example, a single click of a key may cause the user device103to pause the sound. Two clicks may be used to change to a different display device101. The user102may, in some embodiments, hold down a key on the user device103to be able to scan various audio streams, for example, as the user102looks in different directions, or as in a manner similar to the scan function of a car radio.

In this example, a volume slider bar162is provided to enable the user102to control the volume of the audio stream. Alternatively, the user102could adjust the volume using a volume control means built in to the user device103. Additionally, the mute icon158is provided in this screen159to allow the user102to mute and un-mute the audio stream.

In this example, some of the icon control buttons152-157shown inFIG. 7and described above are also shown inFIG. 8. For the screen159, however, only the icon buttons152,153,156and157are shown to illustrate the option of using only those icon control buttons that may be relevant to a particular screen, rather than always using all of the same icon control buttons for every screen.

Furthermore, the screen159includes an ad section163. A banner ad or scrolling ad or other visual message may be placed here if available. For example, the owner of the environment100or the operator of the servers104or other contractors may insert such ads or messages into this screen159and any other appropriate screens that may be used. Additionally, such visual ads or messages or coupons may be provided to the users102via pop-up windows or full screens.

In this example, upon selecting one of the display identifiers160in the display device selection screen159, an additional selection screen may be presented, such as a pop-up window164that may appear over the screen159, as shown inFIG. 9. Some of the video streams that may be provided to the display devices101, for example, may have more than one audio stream available, i.e. may support an SAP (Second Audio Program). The pop-up window164, therefore, illustrates an example in which the user102may select an English or Spanish (Espanol) audio stream for the corresponding video stream. Additionally, closed captioning or subtitles may be available for the video stream, so the user102may turn on this option in addition to or instead of the selected audio stream. The user102may then read the closed captions more easily with the user device103than on the display device102, since the user102may have the option of making the text as large as necessary to read comfortably. Additionally, in some embodiments, the servers104or applications on the user devices103may provide real time language translation to the user102, which may be an option that the user102may select on the pop-up window164. This feature could be stand-alone or connected via the Internet to cloud services such as Google Translate™.

After selecting a desired audio stream and/or closed captioning as inFIGS. 8and/or9, the application may present any appropriate screen while the user102listens to the audio stream (or reads the closed captions). For example, the application may continue to present the display device selection screen159ofFIG. 8or return to the general action selection screen142ofFIG. 7or simply blank-out the screen during this time. For closed captions, a special closed captioning screen (not shown) may be presented. For embodiments in which the environment100is a home or movie theater, for example, it may be preferable to ensure that the screen of the user device103does not put out too much light that might annoy other people in the home or movie theater. The special closed captioning screen, for example, may use light colored or red letters on a dark background, to minimize the output of light. In some embodiments, the screen on the user device103could show any data feed that the user102desires, such as a stock ticker.

While the user102is listening to the audio stream, the user102may move around within the environment100or even temporarily leave the environment100. In doing so, the user102may go out of range of the network access point105that is supplying the audio stream. For example, the user102may go to the restroom in the environment100or go outside the environment100to smoke or to retrieve something from the user's car and then return to the user's previous location within the environment100. In this case, while the user device103is out of range of the network access point105intended to serve the desired audio stream, the corresponding server104may route the audio stream through another server104to another network access point105that is within range of the user device103, so that the user device103may continue to receive the audio stream relatively uninterrupted. Alternatively, the application may present another screen to inform the user102of what has happened. For example, another pop-up window165may appear over the screen159, as shown inFIG. 10. In this example, the pop-up window165generally informs the user102that the network access point105is out of range or that the audio stream is otherwise no longer available. Optionally, the application may inform the user102that it will reconnect to the network access point105and resume playing the audio stream if it becomes available again. Additionally, the application may prompt the user102to select a different audio stream if one is available. In some embodiments, the application may drop into a power save mode until the user102selects an available display identifier160.

In some embodiments, more than one of the network access points105may provide the same audio stream or service the same display device101. Alternatively, the servers104may keep track of which of the display devices101are presenting the same video stream, so that the corresponding audio streams, which may be serviced by different network access points105, are also the same. In either case, multiple network access points105located throughout the environment100may be able to transmit the same audio streams. Therefore, some embodiments may allow for the user devices103to switch to other network access points105as the user102moves through the environment100(or relatively close outside the environment100) in order to maintain the selected audio stream. The SSIDs of more than one network access point105may be the same to facilitate such roaming. This feature may superficially resemble the function of cell phone systems that allow cell phones to move from one cell transceiver to another without dropping a call.

In some embodiments, the application on the user device103may run in the background, so the user102can launch a second application on the user device103. However, if the second application logs into an SSID not associated with the network access points105or servers104for the audio streaming, then the audio streaming may be disabled. In this case, another screen or pop-up window (not shown) may be used to alert the user102of this occurrence. However, if the user device103has already lost contact with the network access point105(e.g. the user102has walked out of range), then the application may allow the changing of the SSID without interference.

An example mode selection screen166for setting a mode of listening to the audio stream is shown inFIG. 11. The application on the user device103may present this or a similar screen when the user102presses the mode icon153, mentioned above. In this example, an enlarged image167of the mode icon153(e.g. an image or drawing of the back of a person's head with wired earbuds attached to the person's ears) is shown in about the middle of the screen166. The letters “L” and “R” indicate the left and right earbuds or individual audio streams. A touch switch168is provided for selecting a mono, rather than a stereo, audio stream if desired. Another touch switch169is provided for switching the left and right individual audio streams if desired. Additionally, the volume slider bar162, the ad section163and some of the icon buttons152,153,156and157are provided. Other embodiments may provide other listening mode features for selection or adjustment or other means for making such selections and adjustments. In still other embodiments, the application does not provide for any such selections or adjustments.

An example equalizer selection screen170for setting volume levels for different frequencies of the audio stream is shown inFIG. 12. The application on the user device103may present this or a similar screen when the user102presses the equalizer icon156, mentioned above. In this example, slider bars171,172and173are provided for adjusting base, mid-range and treble frequencies, respectively. Additionally, the ad section163and some of the icon buttons152,153,156and157are provided. Other embodiments may provide other equalizer features for selection or adjustment or other means for making such selections and adjustments. In still other embodiments, the application does not provide for any such selections or adjustments.

An example settings selection screen174for setting various preferences for, or obtaining various information about, the application is shown inFIG. 13. The application on the user device103may present this or a similar screen when the user102presses the settings icon157, mentioned above. In this example, the username of the user102is “John Q. Public.” An option175is provided for changing the user's password. An option176is provided for turning on/off the use of social networking features (e.g. Facebook is shown). An option177for turning on/off a setting to login anonymously. An option178is provided that may lead the users102to further information on how to acquire such services for their own environments100. An option179is provided that may lead the users102to a FAQ (answers to Frequently Asked Questions) regarding the available services. An option180is provided that may lead the users102to a text of the privacy policy of the owners of the environment100or operators of the servers104regarding the services. An option181is provided that may lead the users102to a text of a legal policy or disclaimer with regard to the services. Additionally, an option182is provided for the users102to logout of the services. Other embodiments may provide for other application settings or information.

For embodiments in which the environment100is a bar or restaurant type of establishment, an initial food and drinks ordering screen200for using the application to order food and drinks from the establishment is shown inFIG. 14. The application on the user device103may present this or a similar screen when the user102presses the “order food and drinks” touch screen button145or the services icon154, mentioned above. In this example, a “favorites” option201is provided for the user102to be taken to a list of items that the user102has previously or most frequently ordered from the current environment100or that the user102has otherwise previously indicated are the user's favorite items. A star icon is used to readily distinguish “favorites” in this and other screens. An “alcoholic beverages” option202is provided for the user102to be taken to a list of available alcoholic beverages. Information provided by the user102in other screens (not shown) or through social networking services may help to confirm whether the user102is of the legal drinking age. A “non-alcoholic beverages” option203is provided for the user102to be taken to a list of available non-alcoholic beverages, such as sodas, juices, milk, water, etc. A “munchies” option204is provided for the user102to be taken to a list of available snacks, hors d′oeuvres, appetizers or the like. A “freebies” option205is provided for the user102to be taken to a list of free items that the user102may have qualified for with “loyalty points” (mentioned above), specials or other giveaways. A “meals/food” option206is provided for the user102to be taken to a list of available food menu items. A “search” option207is provided for the user102to be taken to a search screen, as described below with reference toFIGS. 15 and 16. Additionally, the “Back” (at143) and “Cancel” (at144) touch screen buttons, the mute icon158and the icon control buttons152-157are also provided (mentioned above). Other embodiments may provide for other options that are appropriate for an environment100in which food and drink type items are served.

In this example, if the user102selects the “search” option207, then the user102may be presented with a search screen208, as shown inFIG. 15. Tapping on a search space209may cause another touch screen keyboard (e.g. as inFIG. 6at137) to appear below the search space209, so the user102can enter a search term. Alternatively, the user102may be presented with a section210showing some of the user's recently ordered items and a section211showing some specials available for the user102, in case any of these items are the one that the user102intended to search for. The user102could then bypass the search by selecting one or more of these items in section210or211. Additionally, the “Back” (at143) and “Cancel” (at144) touch screen buttons, the mute icon158and the icon control buttons152-157are also provided (mentioned above). Other embodiments may present other search options that may be appropriate for the type of environment100.

In this example, if the user102enters a search term in the search screen208, then the user102may be presented with a results screen212, as shown inFIG. 16. In this case, the search term entered by the user102is shown in another search space213, and search results related to the search term are shown in a results space214. The user102may then select one of these items by pressing on it or return to the previous screen to do another search (e.g. pressing the “back” touch screen button143) or cancel the search and return to the initial food and drinks ordering screen200or the general action selection screen142(e.g. pressing the “cancel” touch screen button144). Additionally, the mute icon158and the icon control buttons152-157are also provided (mentioned above). Other embodiments may present other results options that may be appropriate for the type of environment100.

In this example, if the user102selects an item to purchase, either from the search or results screens208or212or from any of the screens to which the user102was directed by any of the options201-206on the initial food and drinks ordering screen200, then the user102may be presented with an item purchase screen215, as shown inFIG. 17. A set of order customization options216may be provided for the user102to make certain common customizations of the order. Alternatively, a “comments” option217may be provided for the user102to enter any comments or special instructions related to the order. Another option218may be provided for the user102to mark this item as one of the user's favorites, which may then show up when the user102selects the “favorites” option201on the initial food and drinks ordering screen200in the future. Another option219may be provided for the user102to add another item to this order, the selection of which may cause the user102to be returned to the initial food and drinks ordering screen200. A “place order” option220may be provided for the user102to go to another screen on which the user102may review the entire order, as well as make selections to be changed for the order. Additionally, the “Back” (at143) and “Cancel” (at144) touch screen buttons, the mute icon158and the icon control buttons152-157are also provided (mentioned above). Other embodiments may present other options for allowing the user102to customize the selected item as may be appropriate.

In this example, if the user102chooses to purchase any items through the application on the user device103, e.g. by pressing the “place order” option220on screen215, the user102may be presented with a screen221with which to place or confirm the order. In this example, the user102has selected three items222to purchase, one of which is free since it is perhaps a freebie provided to all customers or perhaps the user102has earned it with loyalty points (mentioned above). The user102may change any of the items222by pressing the item on the screen221. Favorite items may be marked with the star, and there may be a star touch screen button to enable the user to select all of the items222as favorites. Any other discounts the user102may have due to loyalty points or coupons may be shown along with a subtotal, tax, tip and total. The tip percentage may be automatically set by the user102within the application or by the owners/operators of the environment100through the servers104. The user's table identifier (e.g. for embodiments with tables in the environment100) is also shown along with an option223to change the table identifier (e.g. in case the user102moves to a different table in the environment100). Selectable options224to either run a tab or to pay for the order now may be provided for the user's choice. The order may be placed through one of the servers104when the user102presses a “buy it” touch screen button225. The order may then be directed to a user device103operated or carried by a manager, bartender or wait staff person within the environment100in order to fill the order and to present the user102with a check/invoice when necessary. In some embodiments, payment may be made through the application on the user device103to the servers104, so the wait staff person does not have to handle that part of the transaction. Additionally, the “Back” (at143) and “Cancel” (at144) touch screen buttons, the mute icon158and the icon control buttons152-157are also provided (mentioned above). Other embodiments may present other options for allowing the user102to complete, confirm or place the order as may be appropriate.

An example architecture for connecting at least some of the A/V equipment within the environment100is shown inFIG. 19in accordance with an embodiment of the present invention. (Other embodiments in which the functions of the server104are not within the environment100are described elsewhere.) The A/V equipment generally includes one or more A/V sources226, one or more optional receiver (and channel selector) boxes or A/V stream splitters (the optional receiver)227, one or more of the display devices101, one or more of the servers104and one or more wireless access points (WAPs)228(e.g. the network access points105ofFIG. 1). It is understood, however, that the present invention is not necessarily limited to the architecture shown. Additionally, some variations on the illustrated architecture may render some of the components or connections unnecessary or optional.

The A/V sources226may be any available or appropriate A/V stream source. For example, the A/V sources226may be any combination of cable TV, TV antennas, over-the-air TV broadcasts, satellite dishes, VCR/DVD/Blue-ray/DVR devices or network devices (e.g. for Internet-based video services). The A/V sources226, thus, provide one or more A/V streams, such as television programs, VCR/DVD/Blue-ray/DVR videos, Internet-based content, etc.

The optional receivers227may be any appropriate or necessary set top boxes or intermediary devices as may be used with the A/V sources226. The receivers227are considered optional, since some such A/V sources226do not require any such intermediary device. For embodiments that do not include the optional receivers227, the A/V streams from the A/V sources226may pass directly to the display devices101or to the servers104or both. To pass the A/V streams to both, one or more A/V splitters (e.g. a coaxial cable splitter, HDMI splitter, etc.) may be used in place of the optional receivers227.

Some types of the optional receivers227have separate outputs for audio and video, so some embodiments pass the video streams only to the display devices101and the audio streams only to the servers104. On the other hand, some types of the optional receivers227have outputs only for the combined audio and video streams (e.g. coaxial cables, HDMI, etc.), so some embodiments pass the A/V streams only to the display devices101, only to the servers104or to both (e.g. through multiple outputs or A/V splitters). For those embodiments in which the entire A/V streams are provided only to the display devices101(from either the A/V sources226or the optional receivers227), the audio stream is provided from the display devices101(e.g. from a headphone jack) to the servers104. For those embodiments in which the entire A/V streams are provided only to the servers104(from either the A/V sources226or the optional receivers227), the video stream (or A/V stream) is provided from the servers104to the display devices101.

The servers104provide the audio streams (e.g. properly encoded, packetized, etc.) to the WAPs228. The WAPs228transmit the audio streams to the user devices103. Depending on the embodiment, the WAPs228also transmit data between the servers104and the user devices103for the various other functions described herein. In some embodiments, the servers104also transmit and receive various data through another network or the Internet. In some embodiments, a server104may transmit an audio stream to another server104within a network, so that the audio stream can be further transmitted through a network access point105that is within range of the user device103.

An example functional block diagram of the server104is shown inFIG. 20in accordance with an embodiment of the present invention. It is understood that the present invention is not necessarily limited to the functions shown or described. Instead, some of the functions may be optional or not included in some embodiments, and other functions not shown or described may be included in other embodiments. Additionally, some connections between functional blocks may be different from those shown and described, depending on various embodiments and/or the types of physical components used in the server104.

Each of the illustrated example functional blocks and connections between functional blocks generally represents any appropriate physical or hardware components or combination of hardware components and software that may be necessary for the described functions. For example, some of the functional blocks may represent audio processing circuitry, video processing circuitry, microprocessors, memory, software, networking interfaces, I/O ports, etc. In some embodiments, some functional blocks may represent more than one hardware component, and some functional blocks may be combined into a fewer number of hardware components.

In some embodiments, some or all of the functions are incorporated into one or more devices that may be located within the environment100, as mentioned above. In other embodiments, some or all of the functions may be incorporated in one or more devices located outside the environment100or partially on and partially off premises, as mentioned above.

In the illustrated example, the server104is shown having one or more audio inputs229for receiving one or more audio streams, one or more video inputs230for receiving one or more video streams and one or more combined A/V inputs231for receiving one or more A/V streams. These input functional blocks229-231generally represent one or more I/O connectors and circuitry for the variety of different types of A/V sources226that may be used, e.g. coaxial cable connectors, modems, wireless adapters, HDMI ports, network adapters, Ethernet ports, stereo audio ports, component video ports, S-video ports, etc. Some types of video content may be provided through one of these inputs (from one type of A/V source226, e.g. cable or satellite) and the audio content provided through a different input (from another type of A/V source226, e.g. the Internet). Multiple language audio streams, for example, may be enabled by this technique. The video inputs230and A/V inputs231may be considered optional, so they may not be present in some embodiments, since the audio processing may be considered the primary function of the servers104in some embodiments. It is also possible that the social interaction and/or food/drink ordering functions are considered the primary functions in some embodiments, so the audio inputs229may potentially also be considered optional.

For embodiments in which the server104handles the video streams in addition to the audio streams, one or more video processing functional blocks232and one or more video outputs233are shown. The video outputs233may include any appropriate video connectors, such as coaxial cable connectors, wireless adapters, HDMI ports, network adapters, Ethernet ports, component video ports, S-video ports, etc. for connecting to the display devices101. The video processing functional blocks232each generally include a delay or synchronization functional block234and a video encoding functional block235. In some embodiments, however, the sum of the video processing functions at232may simply result in passing the video stream directly through or around the server104from the video inputs230or the A/V inputs231to the video outputs233. In other embodiments, the video stream may have to be output in a different form than it was input, so the encoding function at235enables any appropriate video stream conversions (e.g. from an analog coaxial cable input to an HDMI output or any other conversion). Additionally, since the video streams and audio streams do not necessarily pass through the same equipment, it is possible for the syncing of the video and audio streams to be off by an intolerable amount by the time they reach the display devices101and the user devices103, respectively. The delay or synchronization functions at234, therefore, enable synchronization of the video and audio streams, e.g. by delaying the video stream by an appropriate amount. For example, a generator may produce a video test pattern so that the appropriate delay can be introduced into the video stream, so that the video and audio are synchronized from the user's perspective (lip sync'd).

In this example, one or more optional tuner functional blocks236(e.g. a TV tuner circuit) may be included for a video input230or A/V input231that requires tuning in order to extract a desired video stream or A/V stream. Additionally, for embodiments in which the video and audio streams are received together (e.g. through a coaxial cable, HDMI, etc.), an audio-video separation functional block237may be included to separate the two streams or to extract one from the other. Furthermore, a channel selection/tuning functional block238may control the various types of inputs229-231and/or the optional tuners at236so that the desired audio streams may be obtained. Thus, some of the functions of the display devices101(as a conventional television) or of the optional receivers227may be incorporated into the servers104. However, if only one audio stream for each input229-231is received, then the tuners at236and the channel selection/tuning functions at238may be unnecessary.

The one or more audio streams (e.g. from the audio inputs229, the A/V inputs231or the audio-video separation functional block237) are generally provided to an audio processing functional block239. The audio processing functional block239generally converts the audio streams received at the inputs229and/or231into a proper format for transmission through a network I/O adapter240(e.g. an Ethernet port, USB port, etc.) to the WAPs228or network access points105. Additionally, if it is desired to provide the audio streams to the display devices101as well, then the audio streams may also simply be transmitted through the audio processing functional block239or directly from the audio or A/V inputs229or231or the audio-video separation functional block237to one or more audio outputs241connected to the display devices101.

Depending on the number, type and encoding of the audio streams, some of the illustrated audio processing functions at239may be optional or unnecessary. In this example, however, the audio processing functional block239generally includes a multiplexing functional block242, an analog-to-digital (A/D) conversion functional block243, a delay/synchronization functional block244, an audio encoding (including perceptual encoding) functional block245and a packetization functional block246. The functions at242-246are generally, but not necessarily, performed in the order shown from top to bottom inFIG. 20.

If the server104receives multiple components of one audio stream (e.g. left and right stereo components, Dolby Digital 5.1™, etc.), then the multiplexing function at242multiplexes the two streams into one for eventual transmission to the user devices103. Additionally, if the server104receives more than one audio stream, then the multiplexing function at242potentially further multiplexes all of these streams together for further processing. If the server104receives more audio streams than it has been requested to provide to the user devices103, then the audio processing functional block239may process only the requested audio streams, so the total number of multiplexed audio streams may vary during operation of the server104.

If the received audio streams are analog, then the A/D conversion function at243converts the analog audio signals (using time slicing if multiplexed) into an appropriate digital format. On the other hand, if any of the audio streams are received in digital format, then the A/D conversion function at243may be skipped for those audio streams. If all of the audio streams are digital (e.g. all from an Internet-based source, etc.), then the A/D conversion functional block243may not be required.

Again, since the video streams and audio streams do not necessarily pass through the same equipment, it is possible for the syncing of the video and audio streams to be off by an intolerable amount by the time they reach or pass through the display devices101and the user devices103, respectively. The delay or synchronization functions at244, therefore, enable synchronization of the video and audio streams, e.g. by delaying the audio stream by an appropriate amount. (Alternatively, the audio delay/synchronization functions may be in the user devices103, e.g. as describe below.) For example, a generator may produce an audio test pattern so that the appropriate delay can be introduced into the audio stream, so that the video and audio are synchronized from the user's perspective (lip sync'd). The delay/synchronization functional block244may work in cooperation with the delay/synchronization functional block234in the video processing functions at232. The server104, thus, may use either or both delay/synchronization functional blocks234and244to synchronize the video and audio streams. Alternatively, the server104may have neither delay/synchronization functional block234or244if synchronization is determined not to be a problem in all or most configurations of the overall A/V equipment (e.g.101and103-105). Alternatively, the lip sync function may be external to the servers104. This alternative may be appropriate if, for instance, lip sync calibration is done at setup by a technician. In some embodiments, if the audio and video streams are provided over the Internet, the audio stream may be provided with a sufficiently large lead over the video stream that synchronization could always be assured by delaying the audio stream at the server104or the user device103.

The delay/synchronization functions at234and244generally enable the server104to address fixed offset and/or any variable offset between the audio and video streams. The fixed offset is generally dependant on the various devices between the A/V source226(FIG. 19) and the display devices101and the user devices103. The display device101, for example, may contain several frames of image data on which it would do advanced image processing in order to deliver the final imagery to the screen. At a 60 Hz refresh rate and 5 frames of data, for example, then a latency of about 83 ms may occur.

There are several ways to assure that the video and audio streams are synchronized from the perspective of the user102. One method is to have the user102manually adjust the audio delay using a control in the application on the user device103, which may send an appropriate control signal to the delay/synchronization functional block244. This technique may be implemented, for instance, with a buffer of adjustable depth.

A second method is for the delay/synchronization functions at234and244to include a lip sync calibration generator, or for a technician to use an external lip-sync calibration generator, with which to calibrate the video and audio streams. The calibration may be done so that for each type of user device103and display device101, the application sets the audio delay (via an adjustable buffer) to an appropriate delay value. For instance, a technician at a particular environment100, may connect the calibration generator and, by changing the audio delay, adjust the lip sync on a representative user device103to be within specification. On the other hand, some types of the user devices103may be previously tested, so their internal delay offsets may be known. The server104may store this information, so when one of the user devices103accesses the server104, the user device103may tell the server104what type of user device103it is. Then the server104may set within the delay/synchronization functional block244(or transmit to the application on the user device103) the proper calibrated audio delay to use. Alternatively, the application on each user device103may be provided with data regarding the delay on that type of user device103. The application may then query the server104about its delay characteristics, including the video delay, and thus be able to set the proper buffer delay within the user device103or instruct the server104to set the proper delay within the delay/synchronization functional block244.

A third method is for the server104to timestamp the audio stream. By adjusting when audio is pulled out of a buffer on the user device103, the user device103assures that the audio stream is lip sync'd to the video stream. Each server104may be calibrated for the delay in the video path and to assure that the server104and the application use the same time reference.

A fourth method is for the server104to transmit a low resolution, but lip sync'd, version of the video stream to the application. The application then uses the camera on the user device103to observe the display device101and correlate it to the video image it received. The application then calculates the relative video path delay by observing at what time shift the maximum correlation occurs and uses that to control the buffer delay.

In some embodiments, the video and audio streams may be synchronized within the following specs: Sara Kudrle et. al. (July 2011). “Fingerprinting for Solving A/V Synchronization Issues within Broadcast Environments”. Motion Imaging Journal (SMPTE). This reference states, “Appropriate A/V sync limits have been established and the range that is considered acceptable for film is +/−22 ms. The range for video, according to the ATSC, is up to 15 ms lead time and about 45 ms lag time.” In some embodiments, however, a lag time up to 150 ms is acceptable. It shall be appreciated that it may happen for the audio stream to lead the video stream by more than these amounts. In a typical display device101that has audio capabilities, the audio is delayed appropriately to be in sync with the video, at least to the extent that the original source is in sync.

In some embodiments, problems may arrive when the audio stream is separated from the video stream before reaching the display device101and put through, for instance, a separate audio system. In that case, the audio stream may significantly lead the video stream. To fix this, a variety of vendors offer products, e.g., the Hall Research AD-340™ or the Felston DD740™, that delay the audio by an adjustable amount. Additionally, the HDMI 1.3 specification also offers a lip sync mechanism.

Some embodiments of the present invention experience one or more additional delays. For example, there may be substantial delays in the WAPs228or network access points105as well as in the execution of the application on the user devices103. For instance, Wi-Fi latency may vary widely depending on the number of user devices103, interference sources, etc. On the user devices103, processing latency may depend on whether or not the user device103is in power save mode or not. Also, some user devices103may provide multiprocessing, so the load on the processor can vary. In some embodiments, therefore, it is likely that the latency of the audio path will be larger than that of the video path.

In some embodiments, the overall system (e.g.101and103-105) may keep the audio delay sufficiently low so that delaying the video is unnecessary. In some embodiments, for example, WEP or WPA encryption may be turned off. In other embodiments, the user device103is kept out of any power save mode.

The overall system (e.g.101and103-105) in some embodiments provides a sync solution without delaying the video signal. For example, the server104separates the audio stream before it goes to the display devices101so that the video delay is in parallel with the audio delay. When synchronizing, the server104takes into consideration that the audio stream would have been additionally delayed if inside the display device101so that it is in sync with the video stream. Thus, any extra audio delay created by the network access points105and the user device103would be in parallel with the video delay.

In some embodiments, the video stream may be written into a frame buffer in the video processing functional block232that holds a certain number of video frames, e.g. up to 10-20 frames. This buffer may cause a delay that may or may not be fixed. The server104may further provide a variable delay in the audio path so that the audio and video streams can be equalized. Additionally, the server104may keep any variation in latency within the network access point105and the user device103low so that the audio delay determination is only needed once per setup.

In some embodiments, the overall system (e.g.101,103-105) addresses interference and moving the user device103out of power save mode. In some cases, the delay involved with WEP or WPA security, may be acceptable assuming that it is relatively fixed or assisted by special purpose hardware in the user device103.

If the audio or video delay is too variable, some embodiments of the overall system (e.g.101,103-105) provides alternatively or additionally another mechanism for synchronization. The overall system (e.g.101,103-105) may utilize solutions known in the VoIP (voice over Internet protocol) or streaming video industries. These solutions dynamically adjust the relative delay of the audio and video streams using, for instance, timestamps for both data streams. They generally involve an audio data buffer in the user device103with flow control and a method for pulling the audio stream out of the buffer at the right time (as determined by the time stamps) and making sure that the buffer gets neither too empty nor too full through the use of flow control. In addition or in the alternative, the overall system (e.g.101,103-105) may perform more or less compression on the audio depending on the average available bandwidth.

The audio encoding functions at245(sometimes called codecs) generally encode and/or compress the audio streams (using time slicing if multiplexed) into a proper format (e.g. MP3, MPEG-4, AAC (E)LD, HE-AAC, S/PDIF, etc.) for use by the user devices103. (The degree of audio compression may be adaptive to the environment100.) Additionally, the packetization functions at246generally appropriately packetize the encoded audio streams for transmission through the network I/O adapter240and the WAPs228or network access points105to the user devices103, e.g. with ADTS (Audio Data Transport Stream), a channel number and encryption if needed.

In this example, the server104also has a user or application interaction functional block247. These functions generally include those not involved directly with the audio streams. For example, the interaction functions at247may include login and register functional blocks248and249, respectively. The login and register functions at248and249may provide the screens120,125and134(FIGS. 4,5and6, respectively) to the user devices103and the underlying functions associated therewith for the users102to sign up or login to the servers104, as described above.

In this example, the interaction functions at247may include a settings functional block250. The settings functions at250may provide the screens166,170and174(FIGS. 11,12and13, respectively) to the user devices103and the underlying functions associated therewith for the users102to set various options for the application as they relate to the servers104, including storing setting information and other functions described above. (Some of the underlying functions associated with the screens166,170and174, however, may be performed within the user devices103without interaction with the servers104.)

In this example, the interaction functions at247may include a display list functional block251. The display list functions at251may provide a list of available display devices101to the user devices103for the user devices103to generate the display device selection screen159shown inFIG. 8and the language pop-up window164shown inFIG. 9.

In this example, the interaction functions at247may include a display selection functional block252. When the user102selects a display device101from the display device selection screen159shown inFIG. 8, the display selection functions at252may control the channel selection/tuning functions at238, the inputs229-231, the tuners at236and the audio processing functions at239as necessary to produce the audio stream corresponding to the selected display device101.

In this example, the interaction functions at247may include a content change request functional block253. The content change request functions at253generally enable the users102to request that the TV channel or video content being provided over one of the display devices101to be changed to something different. The application on the user devices103may provide a screen option (not shown) for making a content change request. Then a pop-up window (not shown) may be provided to other user devices103that are receiving the audio stream for the same display device101. The pop-up window may allow the other users102to agree or disagree with the content change. If a certain percentage of the users102agree, then the change may be made to the selected display device101. The change may be automatic through the display selection functions at252, or a manager or other person within the environment100may be alerted (e.g. with a text message through a multifunctional mobile device carried by the person) to make the change. By having the manager or other person within the environment100make the change, the owner/operator of the environment100may limit inappropriate public content within the environment100and may choose video streams that would attract the largest clientele. In either case, it may be preferable not to allow the users102to change the video content of the display devices101(or otherwise control the display devices101) without approval in order to prevent conflicts among users102.

In this example, the interaction functions at247may include a hot spot functional block254. The hot spot functions at254may allow the users102to use the servers104and network access points105as a conventional Wi-Fi “hot spot” to access other resources, such as the Internet. The bandwidth made available for this function may be limited in order to ensure that sufficient bandwidth of the servers104and the network access points105is reserved for the audio streaming, food/drink ordering and social interaction functions within the environment100.

In this example, the interaction functions at247may include a menu order functional block255. The menu order functions at255may provide the screen options and underlying functions associated with the food and drink ordering functions described above with reference toFIGS. 14-18. A list of available menu items and prices for the environment100may, thus, be maintained within the menu order functional block255.

In this example, the interaction functions at247may include a web server functional block256. The web server functions at256may provide web page files in response to any conventional World Wide Web access requests. This function may be the means by which data is provided to the user devices103for some or all of the functions described herein. For example, the web server functional block256may provide a web page for downloading the application for the user devices103or an informational web page describing the services provided. The web pages may also include a restaurant or movie review page, a food/beverage menu, advertisements for specials or upcoming features. The web pages may be provided through the network access points105or through the Internet, e.g. through a network I/O adapter257.

The network I/O adapter257may be an Ethernet or USB port, for example, and may connect the server104to other servers104or network devices within the environment100or off premises. The network I/O adapter257may be used to download software updates, to debug operational problems, etc.

In this example, the interaction functions at247may include a pop ups functional block258. The pop ups functions at258may send data to the user devices103to cause the user devices103to generate pop up windows (not shown) to provide various types of information to the users102. For example, drink specials may be announced, or a notification of approaching closing time may be given. Alternatively, while the user102is watching and listening to a particular program, trivia questions or information regarding the program may appear in the pop up windows. Such pop ups may be part of a game played by multiple users102to win free food/drinks or loyalty points. Any appropriate message may be provided as determined by the owner/operator of the environment100or of the servers104.

In this example, the interaction functions at247may include an alter audio stream functional block259. The alter audio stream functions at259may allow the owner, operator or manager of the environment100to provide audio messages to the users102through the user devices103. This function may interrupt the audio stream being provided to the user devices103for the users102to watch the display devices101. The existing audio stream may, thus, be temporarily muted in order to provide an alternate audio stream, e.g. to announce drink specials, last call or closing time. The alter audio stream functional block259may, thus, control the audio processing functions at239to allow inserting an alternate audio stream into the existing audio stream. Furthermore, the alter audio stream functions at259may detect when a commercial advertisement has interrupted a program on the display devices101in order to insert the alternate audio stream during the commercial break, so that the program is not interrupted.

In this example, the interaction functions at247may include an advertisement content functional block260. The advertisement content functions at260may provide the alternate audio streams or the pop up window content for advertisements by the owner/operator of the environment100or by beverage or food suppliers or manufacturers or by other nearby business establishments or by broad-based regional/national/global business interests. The advertisements may be personalized using the name of the user102, since that information may be provided when signing up or logging in, and/or appropriately targeted by the type of environment100. Additionally, the servers104may monitor when users102enter and leave the environment100, so the owners/operators of the environment100may tailor advertised specials or programs for when certain loyal users102are present, as opposed to the general public. In some embodiments, the servers104may offer surveys or solicit comments/feedback from the users102or announce upcoming events.

Other functions not shown or described may also be provided. For example, the servers104may provide data to the user devices103to support any of the other functions described herein. Additionally, the functions of the servers104may be upgraded, e.g. through the network I/O adapter257.

An example overall network261, in accordance with an embodiment of the present invention, that may include multiple instances of the environment100is shown inFIG. 21. The example network261generally includes multiple environments100represented by establishments262,263and264connected to a cloud computing system or the Internet or other appropriate network system (the cloud)265. Some or all of the controls or data for functions within the establishments262-264may originate in the cloud265.

The establishment263generally represents embodiments in which some or all of the functions of the servers104are placed within the environment100. In this case, the establishment263generally includes one or more of the servers104and WAPs228(or network access points105) on premises along with a network access point266for accessing the cloud265. A control device267may be placed within the establishment263to allow the owner/operator/manager of the establishment263or the owner/operator of the servers104to control or make changes for any of the functions of the servers104and the WAPs228.

The establishment264generally represents embodiments in which some or all of the functions of the servers104are placed within the cloud265. In this case, a server functions functional block268is shown in the cloud265and a router269(or other network devices) is shown in the establishment264. The server functions functional block268generally represents any physical hardware and software within the cloud265that may be used to provide any of the functions described herein (including, but not limited to, the functions described with reference toFIG. 20) for the establishment264. For example, the audio streams, video streams or A/V streams may be provided through, or from within, the cloud265, so the server functions at268process and transmit the audio streams (and optionally the video streams) as necessary to the establishment264through the router269and the WAPs228(or network access points105) to the user devices103(and optionally the display devices101) within the establishment264.

One or more control devices270are shown connected through the cloud265for controlling any aspects of the services provided to the establishments262-264, regardless of the placement of the server functions. For example, software upgrades may be provided through the control devices270to upgrade functions of the servers104or the application on the user devices103. Additionally, the advertisement content may be distributed from the control devices270by the owner/operators of the server functions or by business interests providing the advertisements.

Although embodiments of the present invention have been discussed primarily with respect to specific embodiments thereof, other variations are possible. Various configurations of the described system may be used in place of, or in addition to, the configurations presented herein. For example, additional components may be included in circuits where appropriate. As another example, configurations were described with general reference to certain types and combinations of circuit components, but other types and/or combinations of circuit components could be used in addition to or in the place of those described.

Those skilled in the art will appreciate that the foregoing description is by way of example only, and is not intended to limit the present invention. Nothing in the disclosure should indicate that the present invention is limited to systems that have the specific type of devices shown and described. Nothing in the disclosure should indicate that the present invention is limited to systems that require a particular form of semiconductor processing or integrated circuits. In general, any diagrams presented are only intended to indicate one possible configuration, and many variations are possible. Those skilled in the art will also appreciate that methods and systems consistent with the present invention are suitable for use in a wide range of applications.

While the specification has been described in detail with respect to specific embodiments of the present invention, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing, may readily conceive of alterations to, variations of, and equivalents to these embodiments. These and other modifications and variations to the present invention may be practiced by those skilled in the art, without departing from the spirit and scope of the present invention, which is more particularly set forth in the appended claims.