Apparatus, system, and method for tagging objects in a video stream

An apparatus, system, and method are disclosed for an improved video stream. The method involves taking video of a location and mapping points of interest that are in the range of the camera. Tags are associated with various points of interest and are inserted into the video stream and displayed to the users. Tags may include advertisements and information about particular locations. In certain embodiments, proprietors pay a fee to have a tag associated with their location. The user may be able to remotely control the camera to view a particular area, with the tags appearing at appropriate locations. Video streams from numerous cameras at varying locations may be made available on a website, and end users may select the camera they are interested in and control the camera via the website.

FIELD OF THE INVENTION

This invention relates to inserting information into a video stream, and more particularly relates to inserting advertisements and information into a video stream available over an Internet connection.

BACKGROUND

Travelers are often interested in getting information about destinations prior to their arrival. Various services allow users to survey their destination and acquaint themselves with an area. For example, Google offers a “street view” feature that allows users to see photos of a particular location.

However, many of these services are passive; that is, they show pictures of a location. Many users would prefer to see video of locations of interest. In addition, users may want to find information about the points of interest found within the video frame. Businesses that show up within a frame may want to advertise on sites or channels that show the video stream; other entities, such as government entities and private residences, may want to ensure that their locations are not shown on the video stream.

BRIEF SUMMARY

Presented herein is an approach to providing information to viewers of a video stream. The following summary of certain embodiments is given for convenience, and not by way of limitation.

In one embodiment, the invention is a computer program product comprising computer readable media configured to receive a video stream comprising a plurality of images of a first location taken by a camera at the first location. The computer program product may further comprise instructions for identifying a physical object at the first location that appears in the video stream and identifying a tag that is associated with the physical object. The tag may be an element that provides information about the physical object. The computer program product may also include instructions for inserting the tag into the video stream such that the tag appears to a user receiving the video stream.

Inserting the tag into the video stream may further comprise inserting the tag at a location of the physical object in the video stream. In certain embodiments, the tag is an interactive tag that provides additional information about the physical object when the interactive tag is selected by the user receiving the video stream. The tag may also provide additional information about the physical object in response to a user zooming in the tag or the physical object.

In certain embodiments, the computer program product includes instructions for receiving a plurality of video streams, where each video stream is received from a unique camera at a unique location. It may also receive directional input for the camera from a user of a remote computing device that is receiving the video stream. There may also be instructions for changing the position of the camera in response to receiving the directional input from the user of the remote computing device. The computer program product may also maintain a queue of one or more users and allowing each user in the queue to submit the directional input, wherein only one user is permitted to submit directional input to the camera at a time. Multiple objects may appear in a given video stream, and the computer program product may identify one or more tags that provide information for some of these objects.

In certain embodiments, the computer program product creates a two-dimensional representation of an area viewable by the camera at the first location and associating one or more tags with locations on the two-dimensional representation. In such an embodiment, inserting the tag into the video stream may comprise displaying the tags in the two-dimensional representation over the video stream.

The invention may be realized as a system that includes a first camera at a first location that generates a first video stream comprising a plurality of images of the first location. The system may also include a computer having a processor and memory. The computer may be configured to receive the first video stream from the first camera, identify physical objects that appear in the images of the first video stream, identify tags associated with those objects, and insert the tags into the first video stream such that the tags appear to the users watching the first video stream. The computer may also transmit the first video stream to a second remote computer used by the user watching the first video stream.

The computer may be a server that is connected to the remote computer by an Internet connection. The server may also be remote from the first camera. The system may include multiple cameras at unique locations generating video streams. The computer may receive requests to view one or more of these video streams. The computer may also provide users with a graphical user interface (GUI) that allows the user to select a video stream.

The invention may be realized as a method for providing information in a video stream. The method may comprise receiving a video stream comprising a plurality of images of a first location taken by a camera at the first location and overlaying a two-dimensional map of an area viewable by the camera over the video stream, the two-dimensional map comprising a plurality of tags at locations on the two-dimensional map associated with physical objects, wherein a tag is an element that provides information about the physical object and that appears in the video stream. The method may also involve moving the two-dimensional map in response to changes in a position of the camera such that the plurality of tags appear to users viewing the video stream at locations of associated physical objects in the video stream.

In certain embodiments, the method involves providing additional information about physical objects in response to the user zooming in on at least one of the tag and the physical object. The method may also involve adjusting the position of the tags to account for changes that result from the user zooming in the camera.

DETAILED DESCRIPTION

FIG. 1depicts one embodiment of a system100for an improved video stream. In one embodiment, the system100includes a camera102, a computer104that comprises a video stream apparatus106, a network120, and clients122a-c. Other configurations and systems100other than the example shown inFIG. 1are within the scope of the present invention.

The camera102may be a web camera, an HD video camera, a still camera, or other camera capable of capturing images that be used to create a video stream that are known in the art. The camera102may or may not capture audio information as well as images. The camera102may capture numerous images and transmit them at a frame rate that makes it appear that there is a moving picture. The frame rate and quality of the video stream presented may vary depending on the camera102, the speed of an internet connection, or other factors. Any of various methods for capturing video using a camera102may be used.

In certain embodiments, the camera102may also send information about the position of the camera102and the zoom level of the camera102(referred to hereafter as positional information) to an attached computer104. “Position” refers to where the camera's view is directed. The view is the image being captured by the camera102. The camera102's position may be anywhere within an area of the camera102. This area may be defined by four points: the farthest left and farthest up the camera can direct its view; the farthest right and farthest up that the camera can direct its view; the farthest left and the farthest down the camera can direct its view; and the farthest right and the farthest down the camera can direct its view. Thus, the camera102may be positioned such that its view is a subset of a larger area that can be captured by the camera102if the position and/or zoom level is changed.

In certain embodiments, the camera102may also receive information concerning a desired position and/or zoom level from the computer104. The camera102may be configured to point to the desired position, and/or zoom in or out, based on the information sent by the computer104. The camera102may therefore have the physical machinery necessary to change its position without a user physically manipulating the camera102. The camera102may also have software to interpret the information conveying the desired position that is received from the computer104and adjust its position and/or zoom level accordingly. The information conveying the desired position may originate with the clients122a-c, as described below.

In certain embodiments, the hardware and software necessary to receive and send positional information is physically separate from the camera102; for example, the camera102may be connected to a motorized tripod or other motorized camera holder. In such embodiments, the computer104may send positional information to the tripod and receive positional information from the tripod. The computer104may also be connected to the camera102to send and receive information concerning the zoom level. In certain embodiments, a motorized component physically adjusts the camera102to adjust the zoom level. For example, a motorized component may twist the lens of the camera102or press a button to cause the camera102to zoom in or zoom out.

In certain embodiments, the computer104is configured to provide a digital zoom feature. In such embodiments, the camera102may not have physical zooming capability, or the system100may not use the physical zooming capability of the camera102. Instead, the computer104may be configured to digitally zoom in on the images captured by the camera102. Such an embodiment may relieve the need for the computer104to control the zoom level and receive information about the zoom level of the camera102. In certain embodiments, both a digital zoom and an optical zoom may be used.

In certain embodiments, the camera102is connected to a computer104. The camera102and the computer104may also be physically part of the same device. The computer104may be a laptop, desktop computer, server, or other variety of computer. In certain embodiments, the camera102sends a stream of video that it captures to the computer104. In certain embodiments, the computer104facilitates the transmission of the video stream over a network120to one or more clients122a-c. The computer104may use any of the numerous possible approaches to streaming video over a network120.

In certain embodiments, the computer104includes a video stream apparatus106. The video stream apparatus106modifies the video stream received from the camera102. In certain embodiments, the video stream apparatus106also provides a digital zoom feature. The video stream apparatus106may insert information in the form of tags206into the video stream received from the camera102. The computer104may then feed that video stream over the network120where it can be received by one or more clients122a-c. Known protocols and approaches to streaming video may be used to communicate the video stream over the network120. The video stream apparatus106may also allow clients to adjust the video stream quality and/or frame rate. The video stream apparatus106may allow the user to adjust the frame rate and/or quality; for example, a user with a slow connection over the network120may select a low frame rate or quality.

In certain embodiments, the video stream apparatus106inserts tags that provide information into the video stream. The information may be relevant to the images and/or audio captured by the camera102. In one embodiment, the information comprises advertisements. In certain embodiments, the advertisements are connected to locations or items represented by physical objects within the video stream. For example, the information may include the name and phone number of a business whose building is shown in video stream. The information may also include a coupon for a business that is shown in the video stream. The video stream apparatus106may cause this information to be inserted into the video stream such that it is displayed for users at a location that corresponds to the location of the particular business. This is discussed in greater detail in the example shown inFIGS. 2aand2b.

In certain embodiments, the video stream apparatus106receives positional information from one or more clients122a-c. The video stream apparatus106may thus allow clients122a-cto control the positioning and zoom level of the camera102. In certain embodiments, the clients122a-ccomprise software that allows the users to send positional information to the video stream apparatus106. In other embodiments, the software may be realized in a browser operating on one or more clients122a-c, with the software allowing the users to send positional information being implemented on a remote server. The server may be the computer104, or may be implemented in another location on the network120.

In certain embodiments, a remote server exists on the network120and connects with multiple clients122a-cand multiple computers104, each of which is connected to a camera102. In such an embodiment, the remote server may have access to multiple video streams for multiple locations, and may make those video streams available to multiple clients122a-c. The users of the clients122a-cmay choose which, of the many video streams made available by the remote server, they wish to view. The remote server may also implement a queue or system whereby clients122a-ctake turns controlling the camera102by sending positional information, such that the camera102is under the control of only one client122a-cat any particular time. The remote server may allow each client122a-ca certain amount of time to control the camera102if there are a number of users of clients122a-cthat all wish to control the camera102.

The clients122a-care computers that connect to the network120and are capable of displaying video. The clients122a-cmay be, for example, computers such as laptops, desktops, mobile phones, other mobile devices, televisions, or similar devices that are capable of receiving and displaying a video stream. The clients122a-cmay be operated by a user. In certain embodiments, as noted above, the user may select which video stream to view, and may send a remote camera102positional information and thereby control where the camera102is pointing and what is presented in the video stream.

The network120is a combination of hardware and software that facilitates information sharing between connected entities, such as the clients122a-cand the computer104. The network may comprise gateway devices, routers, servers, Ethernet cabling, and other physical components. The network120may be a LAN, a WAN, or other variety of network120. In one embodiment, the network120provides a connection to the Internet.

WhileFIG. 1shows the video stream apparatus106on a computer104connected directly to the camera102, the video stream apparatus106may also be located at different locations within the system100. For example, the video stream apparatus106may be located on the remote server. In such an embodiment, the remote server may receive the video stream from the computer104that is connected to the camera102and insert tags into the video stream on the remote server. In certain embodiments, the video stream apparatus106may be distributed among a plurality of computers that are in communication with each other in the system100.

FIGS. 2aand2bare exemplary embodiments of an enhanced or augmented video stream. An additional example is shown inFIGS. 7 through 12.FIG. 2ashows one embodiment of a video stream that may be captured by a camera102. In this example, the video stream captures three buildings in the frame202. In the depicted example, a building204is included in the frame202.

The video stream apparatus106may insert information into the frame202for presentation to a user of a client122a-c. For example, the video stream apparatus106may insert a tag206into the frame202. The tag206is an element that appears in the video stream and provides information about a particular item appearing with the video stream. The tag206may be visual, audio, or both. The particular item referenced within the video stream may be visual, audio, or both. In certain embodiments, the tags206are generated by the creators of the video stream apparatus106(i.e., the developers). In other embodiments, the tags206are generated and submitted by users. In certain embodiments, both user-generated and developer-generated content is used in the tags206.

The tag206may provide a variety of information concerning the building204. In the example inFIG. 2a, the tag206shows the name of the business. In certain embodiments, businesses and/or individuals may pay to have tags206appear in the video stream created by a camera102at locations or in connection with events that are associated with their business. For example, the owner of the Zesty Bakery may pay to have the tag206appear in the video stream. In certain embodiments, the revenues from advertisements are split between a company implementing a system (such as system100) and the proprietors of locations that place and maintain the camera102.

The information displayed in the tag206may vary from business to business. In certain embodiments, a business paying for the tag206to appear can dictate what type of information appears in the tag206. The tag206may also change its appearance based on the background in the video stream; for example, the tag206may change color based on the time of day. Thus, white text (as opposed to black text) may be used at night. The tag may be displayed in a color that makes the information easy to read over the background. For example, light text colors are used when dark buildings are in the background. In other embodiments, the color of the tag (as opposed to the color of the text) may change based on the background in the video stream.

The tags206may be placed at or near a particular location captured in the video stream. Such placement may be appropriate when the tag is associated with a physical location, such as a building. Tags206may also be placed based on other considerations. Tags206may be placed to appear over existing advertisements. For example, an advertisement on a billboard may be replaced with a tag206. Similarly, ads on benches may be replaced with tags206. The ads that the system100places on benches and billboards may be for products that do not have an inherent physical location; for example, tags206advertising a particular brand of beverage, or a household cleaner, may be inserted over a billboard.

The tag206may provide a variety of different types of interaction. A user may be able to click on certain tags206to close them. The user may not be permitted to close other tags206, such as those obscuring sensitive locations. Clicking on a tag206may cause the tag206to expand, showing additional information. Clicking on a tag206may open a new browser window on the client122a-cthat opens to the business's website. A tag206may contain a number of hyperlinks which the user can select. A tag206may provide the user with directions from his or her location (or a location of the user's choosing) to the business or location associated with the tag206. The tag206may be equipped to allow the user to print a coupon by clicking on the tag206or a link within the tag206.

Tags206may also be used to obscure sensitive locations that may otherwise appear within a frame202of the video stream. For example, it may be desirable to obscure a government building or a private residence. A tag206may be placed over a sensitive location to prevent the location from appearing within the frame202of a video stream. The tag206may be an advertisement, an image, a box, or other element capable of obscuring the sensitive location. Such a tag206may be referred to as a privacy tag206. The privacy tag206may be inserted into the video stream such that the privacy tag206covers the sensitive object and obscures the sensitive object. By obscuring the sensitive object, the user viewing the video stream cannot see the sensitive object beneath the privacy tag206. The privacy tag206may be, for example, opaque. The privacy tag206may be semi-transparent, but may distort the sensitive object beneath the privacy tag206. Other ways in which the privacy tag206may obscure the sensitive object may also be used. The user viewing the video stream may not be permitted to close the privacy tag206, move the privacy tag206, or otherwise manipulate the privacy tag206to show the underlying sensitive object. The video stream apparatus106may move the privacy tag206in response to a user changing the position of the camera102in order to ensure that the sensitive object is always covered by the privacy tag206.

In certain embodiments, the information displayed in a tag206may also be keyed to a zoom level.FIG. 2bshows a frame210which is zoomed in from the image in frame202. The tag206now displays greater detail about the business associated with the building204. The tag206may display, for example, an address, a phone number, hours of operation, a website address, a menu, or other information.

FIG. 3ashows a second perspective of the scene inFIGS. 2aand2b, but with the camera102zoomed out farther. As can be seen in comparison withFIGS. 2aand2b(andFIG. 3b), the image inFIG. 3ais a superset of that shown inFIGS. 2a,2b, and3b. The view represented inFIGS. 2a,2b, and3bis marked205inFIG. 3a. In addition,FIG. 3ashows how a tag206may be used to obscure a sensitive location. A sensitive location may be any location that the designer or manager of the system providing the video stream determines should not be shown to the viewer. The tag206inFIG. 3amay, for example, be placed over a private residence. The end user may not be given the ability to remove or close the tag206since it obscures a sensitive location. In this manner, privacy issues can be addressed effectively. This approach may further provide additional opportunities for advertising.

FIG. 3bshows one embodiment of a presentation screen310that may be displayed for a user on a client122a-c. In one embodiment, the presentation screen310is part of software that is installed and operating on the client machine122a-c. In other embodiments, the presentation screen310is part of a browser window on the client machine122a-c, and the information about the organization, content, and presentation of the information in the presentation screen310is sent to the browser from a remote server.

In one embodiment, frames202are presented in succession to the user as part of the video stream. Different approaches for presented a video stream may be used. For example, in one embodiment, the video stream is in motion JPEG (MJPEG) format that presents a sequence of separately compressed JPEG images. In another embodiment, the video stream is presented in other formats (such as MPEG-2, JPEG 2000, H.264/MPEG-4 AVC, or others). The tags206are inserted into the video stream. In certain embodiments, the tags206are inserted into the video stream prior to the stream reaching the clients122a-c. In other embodiments, the tags206are not inserted into the video stream directly; rather, the tags206may be superimposed over the video stream. The tags206may then be overlaid on the video stream to present an image such as that shown inFIG. 3b. Other approaches, in addition to the ones explained above, may be used to insert the tags206into the video stream.

Other information and navigation tools may also be presented to a user on a client122a-c. For example, the presentation screen310may provide the user with the ability to control the camera102. In one embodiment, the user may click to zoom, and scroll left-right or up-down to control the camera102. In other embodiments, the presentation screen310may include controls (such as directional arrows and a zoom button) to control the camera102.FIG. 3bshows, at340, examples of controls that a user may use to direct the camera102. The user may be presented with arrows to control the direction of the camera, along with buttons to zoom in and/or out. Other interfaces may be presented to a user to allow the user on the client122a-cto control a camera102.

The presentation screen310may also include a map320. In certain embodiments, the map320contains all points of interest within the range of the camera120. In certain embodiments, only those buildings housing businesses that have paid to advertise appear in the map320. The map320may be interactive; in one embodiment, a user may click on a particular point in the map (for example, the pet store) and the camera102will position itself to capture that point on the map, and present one or more tags206associated with that particular position. This may be an additional way of controlling the camera102.

The presentation screen310may also include a list322that contains categories of businesses, products, and services which may be of interest. In certain embodiments, only those businesses that have paid to advertise appear in the list322. In certain embodiments, if a user selects a particular category, all relevant businesses in that category are listed. For example, selecting “food” may result in all locations within the range of the camera102that offer food being listed. If a user selects a particular location from the list (for example, a particular business name), the camera pans to that location.

In certain embodiments, another panel is shown in the presentation screen310. The panel may display information about what is currently included in the camera120's view. The user may thus easily determine what he or she is looking at. In certain embodiments, if the user positions his or her cursor over an item in the video stream, the name of that particular item appears in the panel.

In certain embodiments, the user may choose to have a tag206disappear. For example, a user may determine that a particular tag206is obscuring another item that the user wishes to examine. The tags206may have an “x” in one corner, which causes the tag206to close if the user selects the “x.” In certain embodiments, the tags206fade and disappear on their own after a given period of time passes. Other approaches to temporarily remove a tag206, or to allow a user to close a tag206, may be used.

FIG. 4shows one embodiment of a video stream apparatus106. In one embodiment, the video stream apparatus106includes a video module410, a mapping module420, a position module430, a control module440, and an information module450. In one embodiment, the video module410receives data from a camera102and prepares the data for transmission to one or more destinations. In one embodiment, the destination is one or more clients122a-c. In certain embodiments, the video module410is responsible for encoding the video stream after additional information (such as tags406) has been added. The video module410may be responsible for inserting the tags206into the video stream based on information provided by the modules described below. The video module410may insert the tags206into the video stream using a variety of approaches. In certain embodiments, the video module410inserts the tags206into the frames that make up the video stream. In certain embodiments, the video module410generates a separate layer to be shown over the video stream, which layer includes the tags206.

The video module410and other modules described below may all be implemented in the same physical location, or distributed through a system100. In certain embodiments, the tags206are inserted into the video stream at the camera102. The tags206may be inserted into the video stream at an intermediate point, such as the computer104or on a server in the network120. For example, a web server on the network120that hosts a website from which a user can select videos may insert the tags206into the video stream. In other embodiments, the tags206are inserted into the video stream at the client122a-c.

The video stream apparatus106may also include a position module430that retrieves position information from the camera102. The position information may include where the camera is pointing in three-dimensional space. In certain embodiments, the position module430receives this information from the camera102. In other embodiments, the position module430may receive position information from a mechanical holder (such as a tripod) that positions the camera102. In certain embodiments, the position module430may communicate with the camera102and a mechanical holder in order to receive position information. As noted above, position information may include a zoom level for the camera102.

The video stream apparatus106may also include a mapping module420that identifies what items are in the video stream and where such items are located in one or more rames of the video stream. In certain embodiments, the mapping module420includes one or more data structures that map particular positions to particular items of interest, such as businesses. In one embodiment, the mapping module420uses a table of positions, where each entry in the table is associated with one or more items of interest. The table may also specify where the items of interest are located in the frame when the camera102is at a particular position.

The mapping module420may also make use of a grid that outlines all possible positions for the camera, and notes where items of interest are located relative to the positions. Other data structures could also be used to associate camera positions with items of interest. Examples of an approach to mapping tags206to the video stream are given below. In certain embodiments, the mapping module420may use software to dynamically identify different items of interest in the video stream. For example, the mapping module420may use a pattern recognition algorithm such as the Burt Pyramid to recognize specific items of interest. In certain embodiments, the mapping module420uses a combination of pattern recognition algorithms and data structures to identify items of interest.

For example, a mapping module420may use data structures to identify relatively static and stationary components of the video stream such as buildings, benches, and billboards. Pattern recognition algorithms may be used to identify dynamic, moving components of the video stream such as cars. Certain advertisements may be tied to dynamic elements; for example, a car manufacturer may want tags to appear when one of their cars drives through the video stream.

The mapping module420may contain the x, y, and z coordinates of all locations that can be seen with the camera102and that should have tag206. In certain embodiments, this information is entered by a user setting up the system100. The mapping module420may also specify the optimal placement of tags206relative to the locations that should have a tag206and other background features.

In certain embodiments, the camera102is fixed to a particular physical location. In such embodiments, the items within the range of the camera102, and the position of items of interest relative to the position of the camera102, are constants. Thus, the mapping module420may indicate where items of interest are located relative to the current position and/or zoom level of the camera102without having to account for the possibility of changes in the location of the camera102itself, and how such changes will affect the locations of items of interest, as stored by the mapping module420. In embodiments which use pattern recognition algorithms, the camera102may not need to be fixed to a particular location. In other embodiments, the mapping module420can account for changes in the location of the camera102and adjust the stored locations of the items of interest accordingly.

The above are examples of ways in which a mapping module420may identify the physical objects that appear in the video stream. In certain embodiments, the physical objects are identified in real-time using pattern recognition algorithms. In certain embodiments, the physical objects are identified when the camera102is initially configured using representations of the area that the camera102can view. The representations may be three-dimensional maps, two-dimensional maps, data structures storing location information, or other approaches to representing an area.

The video stream apparatus106may also include a control module440. In certain embodiments, the control module440allows a user on a client122a-cto re-position the camera102over a remote connection. For example, the user may want the camera102to pan to the left, pan up, or zoom in on a particular point. The control module440may receive control information from clients122a-cand adjust the camera102accordingly. In certain embodiments, the control module440translates the control information received from the clients122a-cinto a format that the camera102can understand and implement. The control information may be received via an intermediate remote server.

In certain embodiments, the control module440maintains a queue of clients122a-cthat wish to control the camera102. In certain embodiments, the control module440allows each client122a-cexclusive control of the camera102for a defined period of time. Once the client122a-chas had control of the camera102for the allotted time, the next client122a-cin the queue is given control of the camera102. All clients122a-cmay be allowed to see the video stream even when the camera102is being controlled by another client122a-c.

The video stream apparatus106may also include an information module450. The information module450may determine what tags206are associated with the items captured by the camera102, and where those tags206should be located in the video stream. In certain embodiments, the information module450uses the position information gathered by the position module430to determine what items (such as buildings) are currently in the frames captured by the camera102. The information module450may also use the mapping module420to determine what tags206are associated with this particular position and to determine where those tags206should be placed in the frame. As discussed above, the tags206may be advertisements, coupons, images for obscuring sensitive locations, or other information. The tags206may also be text, hyperlinks, audio, video, or other type of media. In certain embodiments, the user setting up the system100defines what information appears in the tags206. In this manner, and others, the information module450may identify the tags206that are associated with the physical objects in the video stream.

The video stream is thus presented to users with tags206that may include information, advertising, and valuable content for users of clients122a-c. Users may thus gain greater information about a particular location, and businesses in the vicinity of the camera may be able to reach potential customers. In addition, the user is dynamically engaging with the presented information. This is of particular importance to advertisers. In addition, advertisements placed within the video stream as described above may be insulated from ad-blocking software running on a client122a-c. This approach may therefore be used to ensure more effective delivery of advertising material to users.

FIG. 5is a flow chart diagram showing one embodiment of a method for augmenting a video stream. The steps shown inFIG. 5need not occur in the order shown. The method may begin with determining502a position of the camera102. As noted above, the method may also include determining504the zoom level of the camera102. The method may also include determining506what information is associated with the camera position and the zoom level. For example, as explained above, certain advertisements or information about particular businesses may be associated with the camera102position and zoom level.

The method may also include displaying508the information (such as tags206) associated with the camera position and zoom level. The information may be assigned particular positions in the video stream, which positions are determined to correspond to the information. The correspondence may be made, in certain embodiments, by a developer or a user of the software who associates the information with the location. As shown in the example ofFIGS. 2aand2b, the tag206may be placed at a point that corresponds to the physical location of the business associated with the tag206.

The method may involve determining510whether there has been a change in the camera position or the camera zoom level. If there has been a change, then the information associated with the camera position and zoom level are again determined for the new position and/or zoom level and displayed in the video stream. The method may also involve determining512whether there is any user input. For example, commands from a user that he or she wishes to move the camera to a new position may constitute user input. Other examples include, as mentioned above, a user selecting an item of interest from a map. The method then involves responding514to the user input, which may necessitate finding the information that is associated with a new position and/or zoom level resulting from the user input.

FIG. 6shows one embodiment of a method for capitalizing on an augmented video stream. In one embodiment, the method begins with mapping602locations captured by a camera102. As noted above, this mapping may be done using data structures. In other embodiments, image recognition algorithms are used. The method may also include approaching604proprietors of locations within the range of the camera102and which are captured in the video stream. The method may also involve arranging606for advertisements to appear in the video stream when the particular location is shown in the video stream. In certain embodiments, the advertisement includes details about the business at the location. The advertisement may also include coupons or other advertising materials.

The method may also involve displaying608advertising and information at positions on the video stream that are associated with the camera position and zoom level. Revenues may then be collected610from various businesses and entities that have agreed to pay for advertising in the video stream. In certain embodiments, the revenues from the advertising are shared612with the camera operator. The camera operator may be responsible for setting up and maintaining the camera at the particular location. The camera operator may simply allow another to set up the camera, and may take a portion of the revenues in return for the space provided for the camera.

FIGS. 7 through 12show one embodiment of how an end user on a client122a-cmay interact with a camera102. As discussed above, the end user may use the client122a-cto connection to a website that transmits the video stream captured by the camera102. The website may also provide additional tools to the end user allowing him/her to interact with the camera102. In one embodiment, the video stream captured by the camera102shows the items shown inFIG. 7. The end user may interact with controls on the webpage to control the camera102. For example, the end user may press an “up” control to pan the camera102up. The end user may be presented with corresponding “down,” “left,” and “right” buttons. The end user may also be presented with “zoom in” and “zoom out” buttons.

When the end user encounters an item that has been annotated, a tag206may appear. The tag206may be a visible indicator (as seen inFIG. 8), an audible indicator, some combination of the two, or some other indicator. In certain embodiments, if the end user leaves the camera102stationary for a period of time, the tags206disappear. The period of time may be preset. The tags206may reappear when the user changes the camera view (for example, by pressing one or more controls), or the zoom level.

As the end user increases the zoom level, the information displayed in the tag206may change, as illustrated inFIGS. 9 and 10. For example, the amount of information presented to the end user may increase as the camera102zooms in. In certain embodiments, the information presented in the tag206becomes more and more specific as the camera102zooms in, and becomes more general as the camera102zooms out.FIGS. 11 and 12show a similar change in the granularity of information presented in a tag206as the camera zooms in and out. In certain embodiments, the tag206includes identifying information at every zoom level. Thus, if an end user pans to a new building while the camera102is zoomed in, the end users still sees enough information to identify the building.

A similar approach may be used with tags206that include audio information. A general identification and introduction of a particular location may be provided at a first zoom level. More detailed information may be presented via audio as the camera102zooms in. Other approaches for presenting information to an end user may also be used.

FIG. 13shows one embodiment of an area map1302and a view1304. The area map1302represents the area that the camera102can capture and the location of tags206within that area. The view1304is the portion of the area that the camera102can capture at a given time. The size of the view1304may change relative to the area and the area map1302as the camera102zooms in and out, which makes the camera102capture a view1304that is smaller or larger portion of the area. The area map1302, represented visually inFIG. 13, may be a data structure that allows a computer to relate information about objects in the area map1302with the position of the camera102.

The area map1302may be a layer that is superimposed over the video stream such that the images captured by the camera102and the tags206and other information in the area map1302appear to the user. In another embodiment, the area map1302includes a table of tags206with associated x-y-z coordinates. The video stream apparatus106may determine the x-y-z coordinates of the view1304and consult the table to insert the tags206at the appropriate locations within the video stream that represents the view1304.

In one embodiment, the area map1302represents the area capturable by the camera102as a two-dimensional space. The area map1302may flatten the three-dimensional image captured by the camera102into the two-dimensional representation. The area map1302may associate tags206with particular locations of the area map1302. For example, the house1306aand the house1306bmay be fixed at the locations shown within the area map1302. The area map1302may fix tags206to particular locations (such as the locations of the houses1306a-b) using x-y coordinates or other approaches to associate the tags206with a particular location within the area map1302. In other embodiments, the area may1302is a three-dimensional model with objects associated with particular x-y-z coordinates.

The view1304represents the section of the area that is captured by the camera102. In certain embodiments, as the view1304moves, the changes in position of the camera102causing the movement of the view1304are noted. The video stream apparatus106may determine what sections of the area map1302are currently within the view1304and where they belong within the view1304. The video stream apparatus106may display the tags206at appropriate locations based on where the view1304is within the area map1302.

As noted above, the video module410may insert the tags206directly into the video stream by inserting the tags206into the frames of the video stream captured by the camera102. The video module410may insert the tags206into the video stream by causing the tags206to appear over the video stream shown to the end user. For example, the video module410may cause the tags206to appear over the video stream as it is shown to the user without actually inserting the tags206into the frames.

In one embodiment, the area map1302is implemented as a hidden layer. The video stream apparatus106may display only those tags206within the hidden layer that are also within the boundaries of the view1304. InFIG. 13, the tag206bmay be displayed to the user since it is within the view1304, while the tag206ais hidden from the user since it is outside of the view1304. The area map1302, in such an embodiment, may be curved to more accurately match tags206to objects in the video feed.

The video stream apparatus106may move the area map1302in response to the camera102moving. For example, if the camera102is sent a command to pan left, the video stream apparatus106may move the area map1302by an equivalent amount in the opposite direction from the movement of the camera102. This approach may be used to make it appear to the user that the tags206are matched to the objects (such as the houses1306) in the real-time video stream from the camera102.

To implement such an embodiment, the video stream apparatus106may track the dimensions of the view1304of the camera102and the current X, Y, and Z coordinates of the camera102. Changes to the position of the camera102cause a corresponding change in the position of the view1304. In embodiments where the area map1302is a hidden layer, the hidden layer may be moved relative to the motion of the camera102to given the impression that the tags206are fixed at the positions shown in the video stream.

In certain embodiments, the area map1302adjusts the location of the tags206based on the zoom level of the camera102. A camera102may not stay perfectly centered on an object as it zooms in on a point. As a result, this can cause a tag206to be improperly located as the camera102zooms in. The area map1302may thus be sensitive to zoom levels. In certain embodiments, during setup, the video stream apparatus106zooms in on each object in the area that is associated with a tag206. The video stream apparatus106may take a plurality of samples of the location of the object as the camera102moves from a most zoomed out state to a most zoomed in state. In one embodiment, the video stream apparatus106takes ten samples. The video stream apparatus106may note the change in position of the object at each sample point and store this change information in the area map1302. As a user zooms in on an object, the area map1302may adjust the locations of the tags206within the view1304to account for the changes in position of the objects within the view1304, which changes in position result from zooming in. The video stream apparatus106may also use this depth information in the area map1302in order to determine what additional information should be displayed as the camera102zooms in and zooms out, as described above.

In certain embodiments, the video stream apparatus106uses a plurality of area maps1302that are associated with different zoom levels. As the camera102zooms in, the video stream apparatus106may switch from one area map1302to a second area map1302that is associated with the new zoom level.

FIG. 14shows one embodiment of an area map1302that is implemented as a hidden layer. In certain embodiments, the camera102may have 360 degrees of rotation. As seen inFIG. 14, the video stream apparatus106may cause the area map1302to wrap around such that the full 360 degrees can be represented in the view1304. Thus, when the view1304reaches an edge of the area map1302, the view1304can wrap around to the other side, allowing full representation of the 360 degrees available to the camera102.