Web based video enhancement apparatus, method, and article of manufacture

Controlling an event in a virtual environment is disclosed. A video image from one or more video cameras is received. A video enhanced script is executed to display the virtual environment in a video display within a web browser. An event in the virtual environment is controlled based on the video image. The video enhanced script is configured to communicate with a web browser extension module.

BACKGROUND OF THE INVENTION

1. The Field of the Invention

The present invention relates to networks enabled with digital video, and more particularly to a video plug-in for a web browser.

2. The Related Art

Video teleconferencing is typically implemented by sending video transmissions over standard telephone connections. In the typical video teleconferencing set-up each user has a video camera that transmits an image over standard telephone lines to a monitor at the site of the other user. Video teleconferencing over telephone lines, however, tends to limit users to configurations involving only two sites.

The rise of the Internet and inexpensive video cameras presently allows digitized video images to be transmitted across the Internet or any other wide area network (WAN). Video conferencing over a WAN involves sending a digitized video signal from a video camera to an Internet Service Provider (ISP) connected to the Internet or, more generally, to a server connected to a WAN. The digitized video signal is sent across the WAN to one or more computers that may each display the video signal on a monitor. Information such as digitized video may be transmitted across a WAN by packet switching between servers according to the TCP/IP protocol. This arrangement allows multiple computers to simultaneously share video with one another. Therefore, video conferencing over a WAN has advantages over video teleconferencing in that more sites than just two can be conveniently and inexpensively connected. Video conferencing over a WAN, however, is still limited to sharing real-time images and does not take advantage of image enhancement possibilities.

Other applications for digital video over the Internet and WANs have led to increased sharing of video files and the rise of “webcams.” Video files, pre-recorded video ranging from several seconds to many minutes in length, may be transmitted from a sender to one or more receivers simultaneously across the Internet. A receiver may store the video file and view it repeatedly, provided that the receiver has the necessary software to play the video file. A video file may also be made available, for instance, on a webpage or a FTP site located on a remote server so that multiple users may choose to download it at their convenience.

A webcam is a digital video camera set up to provide live video to the Internet or a WAN such that any user connected to the network may choose to view what the camera is imaging. Webcams are increasingly used to make available, for example, live images of traffic conditions, weather, public meetings, and fish bowls, to users anywhere in the world with Internet access. To simplify the viewing process, some web browsers include a module designed to display live video, so that the user need not run a separate software program. It should be noted that as browsers become increasingly more powerful they become increasingly similar to operating systems. For the purposes of this application, a browser could be an operating system so long as the operating system includes the capabilities typically attributable to browsers such as the ability to connect to remote sites across a WAN and accept software extension modules.

Software extension modules, commonly known as a plug-ins, provide additional functionalities to other software programs. A plug-in, once installed, becomes a seamless part of the software within which it is installed. Plug-in programs may themselves accept additional plug-ins to expand their own capabilities. For example, ShockWave produced by Macromedia, a popular multimedia player plug-in for web browsers, has discrete modules for specific purposes, for example one module for playing sounds and another module for playing animations. ShockWave may be extended with additional plug-ins that act as further modules for other media applications such as live video.

Like text files, sound files, video files, and live video, anything in a digitized format may be sent across a WAN. Consequently, plug-ins are frequently made available over the Internet for downloading from a remote server. Other programs may also be obtained via the Internet. One such program designed to perform a specific operation is called a script. An example of a script is an Applet, a program written in the Java programming language, which can be distributed over a WAN and executed by a Java-enabled web browser. Scripts written in other programming languages may also be distributed over a WAN.

A script may require certain plug-ins to be available in order to perform its function properly. For example, one might design a script to guide a user to select an appropriate automobile for their needs by prompting the user to answer specific questions. As part of the selection process the script might display images of vehicles the user has to choose from. In order to display those images, the script would access the plug-in that enables image viewing and supply that plug-in with file names and locations of the images to be displayed. Those addresses may be on the same computer as the browser, or on a network server that the computer is attached to, or on a remote server accessible over a WAN.

The Internet has also expanded a user's options for communicating with others, in addition to providing a means for accessing programs and data from remote sites. Accordingly, a person can send and receive electronic messages (“e-mail”) over a WAN with typically only a brief time delay. Electronic messages may also be shared in real-time between individuals connected to the Internet; this process has come to be referred to as “chat” and is frequently conducted in “chatrooms,” that are dedicated websites where multiple individuals can converse simultaneously. A chatroom is the functional equivalent of the telephone party-line, except that the conversation is carried out with text rather than with voice.

As communications technologies have been developed for the Internet those technologies have been applied to gaming. It is now possible to download gaming software from a website and play it on a host computer. It is also possible to play a game over the Internet where the gaming software is located on a remote server rather than on the user's computer. Further, multiple players in multiple locations may take part in the same game over the Internet.

In addition to developments concerning the Internet, the recent development of low-cost image sensors used in cameras has led companies to develop more products that include imaging. For instance, Nintendo offers a black and white camera that attaches to a Game Boy personal video game system and a paint program that allows a user to manipulate and alter the images. The Game Boy camera has a lens that rotates so the user can obtain self-portraits. Mattel Media, too, offers a Barbie Digital Camera that takes color images and comes with software that lets children edit their images into Barbie scenes and to further print them on cards and stickers.

Further advancements have also been made in image analysis and recognition. For example, Scientific American reported on the development of a system called Person Finder that can track one person as he or she moves around in a room (“Smart Rooms,” Scientific American, April 1996). The system records the image of the person and the room, determines where the person is, creates a virtual model of the person, and projects the model into a virtual world so that, for instance, imaginary characters can interact with the smart-room user.

Absent from this mix of digital technologies is software that will allow one or more live video images from one or more personal digital video cameras to be incorporated into a virtual environment displayed within one or more browsers so that the person or people can use their image or images to control events within the virtual environment.

SUMMARY OF THE INVENTION

The present invention describes a video enhancer for a web browser. The video enhancer consists of a library of video enhancement routines, a browser extension module coupling the library of routines to the web browser, and a video enhanced script coupled to the browser extension so that it may utilize at least one of the enhancement routines within the library to provide an enhanced video display within the web browser. In some embodiments of the present invention the library of routines are provided as dynamically linked libraries (DLLs) such as C++ DLLs. In some embodiments the browser extension module includes a translator and a multimedia extension module, such as the ShockWave plug-in produced by Macromedia. Other embodiments may be fully integrated to be able to interface directly with a web browser and not need the multimedia extension module.

The video enhancer of the present invention includes a video enhanced script that may be an Applet running in conjunction with the operating web browser or running independently. The video enhanced script calls upon enhanced video routines. Some routines, for instance, provide the ability to identify and track the head of a person in front of a video camera, while other routines may provide the ability to determine when at least two tracked features collide.

Another embodiment of the present invention is a computer-implemented system for playing a game. This computer-implemented system includes a first computer, a first video camera coupled to the first computer, a web browser operating on the first computer, and a video enhancer for the web browser. The video enhancer further includes a library of video enhancement routines that may be in communication with the first video camera, a browser extension module coupling the library to the web browser, and a video enhanced script. The video enhanced script is itself coupled to the web browser and may use at least one enhancement routine of the plurality of routines in the library to provide enhanced video displays within the web browser. Additional embodiments of the computer-implemented system also include a wide area network (WAN) and a server connected to the WAN, with the first computer being connected to the WAN.

In some embodiments of the computer-implemented system the video enhanced script allows the game to be played while the video enhanced script is being downloaded to the first computer from the server via the WAN. In other embodiments the video enhanced script allows a player to send information, such as a cookie, to a server to be retained for future reference after a game is over. In further embodiments the video enhanced script maintains communication with a remote server through the operating web browser. Yet other embodiments of the computer-implemented system further include a second computer that may include a video camera, and is configured like the first computer, though it need not include the video enhancer if the first computer is so equipped.

Another embodiment of the present invention is a method for video enhancement over a wide area network. The method includes providing a web browser, providing a video enhancer for the web browser, opening the web browser, and executing a selected video enhanced script. In some embodiments the method for video enhancement also includes, before executing the script, determining whether a more current version of the video enhancer is available. If an update is available the user is queried whether the more current version should be obtained, and if the user requests the update it is obtained. In some embodiments obtaining the video enhancer update includes downloading the update from a server connected to a WAN and installing the update within the web browser.

In some embodiments of the method for video enhancement the process of executing the selected script further includes passing a command from the selected script to the browser extension to be executed, where the command specifies at least one video enhancement routine and includes parameters required by those routines. In other embodiments executing the selected script instead involves passing a command from the selected script to a multimedia extension module, passing the command from the multimedia extension module to a translator, and executing at least one video enhancement routine.

A further embodiment of the present invention is an article of manufacture comprising a computer readable medium containing code segments for providing a video enhancer for a web browser. The video enhancer includes a library of at least one video enhancement routines and a browser extension module operative to couple the library to the web browser. In some embodiments the video enhancer further includes at least one video enhanced script that may be coupled to the web browser to utilize at least one video enhancement routine.

These and other aspects and advantages of the present invention will become more apparent when the Description below is read in conjunction with the accompanying Drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1illustrates an overview of the basic elements of a system of computers joined across a wide area network. The entire system1consists of a wide area network (WAN)2, such as the Internet, connecting service providers3and allowing the sharing of data packets according to the TCP/IP protocol. Connected to WAN2is a service provider3that is further connected to a server4that is configured to both distribute and receive files, data, and programs across the WAN2. A computer system5of a user is connected through a service provider3′ to the WAN2in order to share files, data, and programs with server4. Additional computer systems5′ may similarly be connected to the WAN2. Each computer system5comprises at least an operating system6, storage media7for saving data, files, and programs, and a web browser8to facilitate communication between the computer system5and the service provider3′.

FIG. 2shows a flow chart for the process10of obtaining a video enhancer of the present invention from a server4. In other embodiments, described below, the video enhancer of the present invention may be obtained on a CD-rom, magnetic storage disk, or it may be provided as part of a package of other software. In the first act or operation12a web browser8is opened. In the next act or operation14the user determines whether the video enhancer is already present within the web browser8. Act or operation14may be prompted by the user's desire to obtain the video enhancer because of a pre-existing awareness of its desirability. Alternately, the user may attempt to run an application within the web browser8environment that requires the video enhancer, however that video enhancer is missing and the user is therefore prompted to obtain the video enhancer. If the video enhancer is present within the web browser8, then the process10is finished. Otherwise, the user progresses to act or operation16.

In act or operation16the web browser8is connected to a server4. Connection to the server4is made by creating a connection to a WAN2such as the Internet and by providing the web browser8with the IP address of the server4. The IP address of the server4may be obtained in many ways. For example, the IP address may be specified by an existing application that will not run without the required video enhancer, it may be obtained by performing a search of the World Wide Web using a search engine, it may be obtained from promotional materials, or it may be obtained from other users. Once the web browser8has connected to the server4, in act or operation18the desired video enhancer is downloaded to the user's computer system5to be stored on a user specified storage medium7. That medium7may be the working memory of the user's computer5, a storage device connected to the computer5such as a magnetic disk drive, or a storage device connected to a network of which the user's computer5is a part. Once the video enhancer is saved to a storage medium7the process10is complete.

FIG. 3shows a flow chart for the process20of selecting and executing a script. In the first act or operation22the user opens a web browser8. In the next act or operation24the user accesses a server4configured to store one or more scripts and selects a script for one or more desirable functionalities. For example, a script may have the functionality of performing a game.

In act or operation26the selected script looks to the user's web browser8and to the server4to determine if the server4has a more current (updated) video enhancer than is present within the user's web browser8. If the video enhancer in the user's web browser8is current, then the script is downloaded in act or operation32. If an updated video enhancer is available on the server4, the user is asked in act or operation28whether she would like the updated video enhancer. If the user requests the updated video enhancer, then it is downloaded from the server4to the user's web browser8and installed therein in act or operation30, prior to downloading the selected script in act or operation32. Alternately, if the user does not want the updated video enhancer at act or operation28then the selected script is downloaded in act or operation32.

In act or operation34the selected script is executed, as will be described in greater detail below with reference toFIG. 3. Once the selected script has been executed, the user has the option in act or operation36to execute the script again. If the user so chooses, act or operation34is repeated, otherwise the process20is complete.

FIG. 4is a schematic diagram of a computer system5including a video enhancer42for a web browser8, an operating system6, and a wide area network (WAN)2. The video enhancer42further includes a library of video enhancement routines50, a browser extension module52, and a video enhanced script54. Both the browser extension module52and the library of video enhancement routines50may be obtained by downloading them from a remote server4over the WAN2to the user's computer system5. Alternately, both the browser extension module52and the library of video enhancement routines50may be downloaded to a local server4to which the user's computer system5is connected. In another embodiment either or both of the browser extension module52and the library of video enhancement routines50may be maintained on a remote server4and accessed over the WAN2.

The operating system6provides a basic framework within which all of the accessories on the user's computer5may operate, including the web browser8and a video camera60. Examples of operating systems6include Apple's operating system for the Macintosh line of computers, Microsoft's Windows 98, and Linux. The web browser8, when coupled with the browser extension module52, becomes a program capable of executing a video enhanced script54. Examples of web browsers44include Microsoft's Internet Explorer and Netscape's Navigator. In order to execute a video enhanced script54, the browser extension module52may run one or more video enhancement routines from the library50, which may, in turn, require the video output from the video camera60. It should also be noted that although the output from a video camera60may at times be preferable, any digitized video image can be used. For example, AVI and MPG files, representing two different formats for digitized video files, or streaming video received across a WAN2may be substituted for the output from a video camera60. It is likewise unimportant where the video image comes from, so the video camera60supplying the image may be linked to computer system5by WAN2, or it may come from a stored video file on a remote server4.

The browser extension module52may be a stand-alone program, however, in other embodiments of the present invention the browser extension module52is comprised of a multimedia extension module56and a translator58. In such an embodiment the multimedia extension module56may be a commercial program such as ShockWave produced by Macromedia. Such multimedia extension modules56are ordinarily capable of interfacing with the operating system6, the web browser8, and the WAN2, but may not be able to communicate with the library of video enhancement routines50. Consequently, when browser extension module52includes a multimedia extension module56it also must include a translator58to facilitate communication between the multimedia extension module56and the library of video enhancement routines50.

FIG. 5shows a flow chart for the process70for executing a video enhanced script54where the browser extension module52includes a multimedia extension module56and a translator58. In the first act or operation72the multimedia extension module56receives a command from the video enhanced script54to run a particular video enhancement routine from the library of routines50. Included in the command from the video enhanced script54should be any parameters required by the video enhancement routine. The multimedia extension module56sends the command to the translator58where it is parsed according to a protocol that specifies data structure.

In act or operation74the parsed command is translated into code that is recognizable by the video enhancement routine. The translated command is then passed to the video enhancement routine. The video enhancement routine next executes the command in accordance with the provided parameters in act or operation76. For example, the video enhanced script54might involve a game for practicing soccer in which the player stands in front of a video camera60and sees herself on a monitor in a virtual world in which she's standing on a soccer field. In this virtual world seen on the monitor soccer balls drop from the sky and the object is for her to move her head so as to “head” the soccer balls towards a net.

This script54might call first for a determination of where the girl's head is located and then call for a determination of whether it has collided with a soccer ball. The multimedia extension module56first receives a command from the script54to execute a video enhancement routine for head tracking, with the command including necessary parameters, for instance a clock signal or a frame refresh rate. The video enhancement routine for head tracking receives this command from the translator, which might request, for example, a determination of head location, contour, and orientation. The head tracking routine might then make these determinations by applying image recognition techniques to the digitized video data.

Similarly, script54might then ask a collision detection routine to determine whether the girl's head has collided with a virtual ball that script54has generated. The command for collision detection would include parameters necessary for the routine to make the determination, such as the data previously returned from the head tracking routine. Next, the output from this video enhancement routine, for instance that the head and the ball had collided and the points of contact on each, is sent back to the translator58to be parsed in act or operation78. Then, in act or operation80this output is translated into code having a data structure that is recognizable by the multimedia extension module56, and passed back to the multimedia extension module56.

FIG. 6shows a flow chart for the process90for downloading and executing a video enhanced script54. The process begins with act or operation92in which a script54is downloaded to the user's computer5. Downloading is accomplished by opening a web browser8, accessing a server4on which the script54resides, and copying the script54to the computer5on which the web browser8resides. Accessing the server4is achieved by furnishing the web browser8with the server's IP address, thus allowing the web browser8to contact the server4across a WAN2such as the Internet. Copying the script54to the user's computer5involves having the server4on which the script54is stored divide the script54into packets that can be routed across the WAN2to be reassembled by another server4in communication with the user's computer5. The script54, when received by the user's computer5, can then be stored, for example, in RAM, on a magnetic recording medium7, or on a server4in communication with the computer5.

In act or operation94the script54asks whether a cookie is desired. If a cookie is desired it will be obtained in act or operation96from a server4, such as the one that delivered the script54. The cookie will then be stored either in RAM or on a storage device such as a hard drive7. Cookies may be used to record one's activities, for example, when accessing a website. Cookies also may record personal information, such as passwords and personal preferences, so the when a user returns to a site, the user may be presented with customized information.

Once a cookie has been obtained, or if no cookie was sought, the video enhancer42runs script54in act or operation98, as described above with reference toFIG. 3. The video enhancer42in act or operation100then displays the results. The results may be displayed on one or more monitors, where some monitors may be connected to the user's computer5via a WAN2. The displayed results may, for example, show a continually refreshed video of the field of view of a video camera60with an animated feature superimposed. For instance, a person standing in a room before a video camera60might see on a monitor an image of himself in the room with a gremlin hopping on his head. In other embodiments the results may show an actual image of a person in the field of view of a video camera60against a virtual backdrop. The person in the above example might see a true image of himself against the backdrop of a cartoon room, or an outdoor scene, or in the midst of an animated cartoon, a non-animated movie segment, or a music video. In further embodiments the displayed results may show an actual image of a person in the field of view of a video camera60with superimposed animated body parts or accessories such as a hat or a sword. Other embodiments may combine elements described above. For example, a person standing before a video camera60might see himself with superimposed chaps, hat, and holster against a background that's a still image of the OK Corral with animated tumbleweeds rolling past in the distance.

Yet other embodiments of the present invention envision multi-player games in which players interact from different locations across a WAN2. In the above mentioned example of the person seeing himself at the OK Corral, a second player standing in front of a second video camera60could appear in the first person's image as an adversary. Here, the two players might be either be communicating directly across a WAN2, or they may be communicating across a WAN2through a remote server4not directly linked to either players' computer5. In some instances it may be desirable to provide cartoon renderings of remotely located players, rather than real-time images, to reduce the amount of data that must be transmitted. In other cases a real-time image that is not refreshed as frequently may be used, and in other cases only a part of the body, for example the player's head, may be a real-time image while the remainder of his body might be a cartoon rendering.

When a video enhanced script54has displayed the final results in act or operation100, the user may be prompted by the script54in act or operation102whether the script54should be executed anew, which would return the process90to act or operation98. If the script is not to be repeated, then in act or operation104the script asks whether a cookie should be stored. If a cookie is to be stored, that cookie will be stored to a local storage device7in act or operation106.

FIG. 7shows a flow chart for the process110performed by a server4connected to one or more computer systems5across a WAN2. In act or operation112a user establishes a connection to the server4by providing the IP address of the server4to the web browser8of the user's computer system5. Once the connection is established, the server4may ask the user to log in by providing a password or some other identifying characteristics. In this way user's may be identified to the server4as belonging to one of several possible groups such as paid members, holders of trial subscriptions, and first-time visitors. The group to which the user belongs may limit the types of services the server4may perform for that person, or it may prompt the server4to issue special offers or run particular banner ads within the user's web browser8.

In act or operation114the server4determines whether a browser extension module52exists within the web browser8, and if so, whether it is the most current version available. This sort of information may already exist on the server4in the form of a cookie. Alternately, the server4may ask for the information from the web browser8. If the server4has a browser extension module52update available the process proceeds to act or operation116, otherwise it progresses to act or operation120. In act or operation116an updated browser extension module52is made available to the user. If the user decides to download the updated module52this is accomplished in act or operation118. If the user decides to forego the update then the process proceeds to act or operation120. It should be noted that server4may represent a plurality of linked servers, so that the server4supplying the browser extension module52may not necessarily be the same server4that is providing video enhanced script54. A system of linked servers are able to provide greater capacity to handle large volumes of requests and provide redundancy so that if one server should fail others will take up the load.

In act or operation120the server4processes requests from the user by determining what is being asked for and then initiating a task in response. As the server4processes a request in act or operation120it also records in a database information about the request, for example, the identity of the user and what was requested. A common occurrence might be where the user's computer system5requests files or data from the server4, or seeks to upload files or data to the server4, which would be performed by act or operation122. This situation may occur where a user is involved in a game with another user and video images of both users are being passed between the users via the server4.

Similarly, the user may request in act or operation120that the server4provide a copy of the browser extension module52or of a video enhanced script54for downloading to the user's computer system5. These tasks are handled, respectively, by act or operation124and act or operation126.

In act or operation128the server4handles requests for IP addresses, for example when a user requests to be connected directly to another user. In handling such a request the server4may need to consult a database to determine whether the user is a person with authorization to obtain the requested IP address.

In act or operation130the server4handles requests to connect users to other users through the server4. For instance, a user may wish to join a game, but is unaware of any other users presently involved in a game and willing to have others join. To process such a request the server4may have to consult a database listing games currently in progress and providing their status. Additional information in such a database may include cross-referenced lists of people that would indicate certain users that should not be connected to other certain users.

In act or operation132the server4handles requests requiring secure access. In these requests a system administrator, for example, would have to provide a special password to be able to control and observe system data in act or operation134. A system administrator, may, for example, access sensitive databases, upload new versions of software, or monitor games in progress.

In act or operation136the user is offered the choice of logging off of the server4or making another request of the server4. If the response is to stay connected and make a further request, the process110returns to act or operation120, otherwise the process110is completed.