Scalable automated empirical testing of media files on media players

Scalable empirical testing of media file playback utilizes test hooks in each media player to support simulated human interaction and playback monitoring. A media crawler catalogs media files accumulated in a media file database to create a wordlist. One or more scalable instances of media tester accesses the wordlist to select items of work linked to media files. Work items and/or operating modes of media tester specify test parameters such as performance profiles or further define testing such as specifying repetitious playback on one or more media players. Media files are downloaded to and played by a scalable number of media players. Playback performance is monitored, analyzed and reported. Failure reports are accompanied by instructions to reproduce failures and cross-references to content or source code in media files. Failures can be audited by additional work items for follow-up testing.

TECHNICAL FIELD

The technical field relates generally to automated testing and, more specifically, to scalable automated empirical testing of media files on media players.

BACKGROUND

Media players such as game consoles, computers, video players, and cellular telephones are usually capable of playing a number of media files such as movies, music and pictures stored in multiple formats. Media players and media files are usually created independent of one another, although with generic compatibility in mind. However, compliance with standards and specifications is not always sufficient to guarantee compatibility or the absence of defects. To guarantee accurate playback of media files on media players, some form of empirical testing may be necessary to detect any incompatibilities or defects in the media files and/or media players.

For quality assurance purposes, it is desirable for voluminous content service providers such as Microsoft Corporation to ensure that the many media files it serves to customers, e.g., through Internet accessible XBOX LIVE and XBOX LIVE Marketplace located at http://www.xbox.com, will playback properly on Microsoft's XBOX 360 gaming console or other supported media player. XBOX, XBOX 360, and XBOX LIVE are registered trademarks of Microsoft Corporation, One Microsoft Way, Redmond, Wash. 98052-6399. XBOX LIVE is a full spectrum online gaming and entertainment service. Besides providing online multiplayer gaming, through XBOX LIVE and XBOX LIVE Marketplace, customers can download purchased and promotional content to their XBOX 360 hard drive, computer or other media player, including high definition and standard definition television shows, movies, gaming videos, music videos, short feature films, video games, dashboard themes, slideshows, gamer pictures, game trailers/demos, movies, game content such as new maps, weapons, levels, characters, challenges, expansions, arcade games, demos and trailers. The thousands of media files available on XBOX LIVE may be produced by Microsoft and many other companies.

While human screeners may review playback of media files on various playback devices such as the XBOX 360, reliance on humans has inherent limitations, including physical and mental fatigue, forgetfulness, and inconsistency. For example, a human screener may doze off or otherwise fail to pay attention periodically during playback or may forget to pause playback during a break. Humans may also suffer eye strain while critically reviewing media file playback hours on end. Further, perceptions of playback may vary from one human to the next. Further still, time and budgetary constraints make it nearly impossible to review all media files, especially after every hardware, firmware or software improvement to playback devices. In short, as the volume of media content grows, it becomes more and more infeasible to manage content validity. Thus, at present, too much reliance on quality assurance is left to customer reporting.

SUMMARY

Scalable empirical automated testing of voluminous media files on media players is accomplished by utilizing test hooks in each media player and/or in accessory devices such as game controllers or remote controls communicatively coupled to each media player to support simulated human interaction and playback monitoring. The scalable test environment may be centralized or distributed. Media files requiring testing may be accumulated in a media file database. A media crawler may catalog media files stored in the media file database to create a wordlist. A scalable number of instances of a media tester program may access the wordlist to select items of work linked to media files. Selection of work items may be done randomly or by checkout to avoid collisions with other instances of the media tester program. Work items and/or operating modes of media tester may specify test parameters such as performance profiles or further define testing such as by specifying repetitious playback on one or more media players. Media files are downloaded to and played by a scalable number of media players. Simulated human interaction with media players to play media files may involve simulating human inputs necessary to load media files and navigate menus. Media applications such as games may be accompanied by supporting information used during testing to simulate human interaction with an input device such as a game controller or remote control in order to simulate, for example, a human playing a movie or video game. Media file playback or media application execution performance may be monitored, analyzed and reported. Failure reports may be accompanied by instructions to reproduce failures and cross-references to content and/or source code. Failures may be audited by follow-up testing.

Empirical testing offers very useful real-world feedback because testing closely follows scenarios that customers follow while playing media files or media applications. Automated empirical testing navigates the same user interface, triggers the same code paths on the same hardware and results in the same memory footprints experienced by media players used by customers. As such, empirical test results are more realistic and useful than emulation test results. A scalable empirical testing system with ones, tens, hundreds or more media players operating 24 hours per day 365 days per year permits content providers to process large volumes of media files in short periods of time, which permits greater flexibility in making software improvements to media players. A scalable empirical testing system permits testing throughout development and production of hardware and software builds of media players. A scalable empirical testing system also removes the burden of testing from customers, increases the level of confidence customers have in downloaded content and, ultimately, improves the quality reputation of content service providers having such a system.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Reference will now be made in detail to embodiments of the present technology for scalable automated empirical testing of media files on media players, examples of which are illustrated in the accompanying drawings. While the technology for scalable automated empirical testing of media files on media players will be described in conjunction with various embodiments, it will be understood that they are not intended to limit the present technology for scalable automated empirical testing of media files on media players to these embodiments. On the contrary, the presented technology for scalable automated empirical testing of media files on media players is intended to cover alternatives, modifications, and equivalents, which may be included within the spirit and scope the various embodiments as defined by the appended claims. Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present technology for scalable automated empirical testing of media files on media players. However, the present technology for scalable automated empirical testing of media files on media players may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present embodiments.

Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present detailed description, discussions utilizing terms such as “opening”, “determining”, “sequencing”, “reading”, “loading”, “overriding”, “writing”, “creating”, “including”, “comparing”, “receiving”, “providing”, “generating”, “associating”, and “arranging”, or the like, refer to the actions and processes of a computer system or similar electronic computing device. The computer system or similar electronic computing device manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices. The present technology for scalable automated empirical testing of media files on media players is also well suited to the use of other computer systems such as, for example, optical and mechanical computers. Additionally, it should be understood that in embodiments of the present technology for scalable automated empirical testing of media files on media players, one or more of the steps can be performed manually.

Overview of Terminology

The following terms should be considered in light of the following overviews provided for them as well as the context in which they appear.

Media file: computerized information accessible by one or more media players or software programs running thereon that is used by a media player to generate audible, visual or other sensory information.

Media content: a media file or media application/program such as a video game.

Media player: any device capable of accessing one or more media files and having at least one integrated audible, visual or other sensory presentation device or at least one output thereto. Media Player includes, but is not limited to, fixed or portable game consoles, computers, audio and/or video players, cellular telephones. Some media players, such as XBOX 360, are able to access media content, e.g., play media files and execute media applications.

Failure: Any unintended sensory presentation or other performance flaw falling outside a desired performance profile relating to access to a media file by a media player. A failure may occur in media player hardware, firmware, operating system, application, etc. or in media file or media application.

Scalable empirical automated testing of voluminous media files on media players is accomplished by utilizing test hooks in each media player and/or in accessory devices such as game controllers or remote controls communicatively coupled to each media player to support simulated human interaction and playback monitoring. The scalable test environment may be centralized or distributed. Media files requiring testing may be accumulated in a media file database. A media crawler may catalog media files stored in the media file database to create a work item list. A work item list can comprise list of test cases that can be stored in a database or the like (e.g., SQL database) for query access. One or more instances of a media tester program may access the wordlist to select items of work linked to media files. Selection of work items may be done randomly or by checkout to avoid collisions with other instances of the media tester program. Work items and/or operating modes of media tester may specify test parameters such as performance profiles or further define testing such as by specifying repetitious playback on one or more media players. Media files are downloaded to and played by one or more media players. Simulated human interaction with media players to play media files may involve simulating human inputs necessary to load media files and navigate menus. Media applications such as games may be accompanied by supporting information used during testing to simulate human interaction with an input device such as a game controller or remote control in order to simulate, for example, a human playing a video game. Media file playback or media application execution performance may be monitored, analyzed and reported. Failure reports may be accompanied by instructions to reproduce failures and cross-references to content and/or source code. Failures may be audited by follow-up testing.

FIG. 1is a block diagram of an exemplary system in which scalable automated empirical testing of media files on media players can be implemented. Numerous components have been simplified or omitted fromFIG. 1in order to focus on selected features of scalable automated empirical testing of media files on media players. In the embodiment shown inFIG. 1, empirical testing system100comprises at least one instance of media tester110, a scalable number of media players120, media crawler125, media file store130and test database135. Media players120is referenced in the singular, as in media player120as well as in the plural media players120. It is to be understood that the configuration of the scalable empirical testing system100depicted inFIG. 1is not limited thereto. For example, each of the components of the scalable empirical testing system100can comprise a single processor or multiple processors. And, each of the components can be configured as a centralized component and/or a distributed component. Further, the scalable empirical testing system100can be configured as a centralized computing environment and/or a distributed computing environment.

Media tester110comprises at least one instance of a media tester program executed by one or more computer systems such as the generic computing system600shown inFIG. 6. Media crawler125comprises at least one instance of a media crawler program executed by one or more computer systems such as the generic computing system600shown inFIG. 6. In some embodiments media crawler125may be merged with media tester110in a single computer system. Media players120comprises a scalable number of computer systems, such as the generic computing system600shown inFIG. 6, or other media players capable of playing media files used for testing. For example, as shown inFIG. 1, media players120comprises a plurality of XBOX 360 game consoles, each with an accompanying game controller as a user interface, which is described in greater detail with regard to the exemplary game console500shown inFIG. 5. Media players120may comprise any number of media players, each of which may be the same or different. For example, in some embodiments there may be multiple banks of media players, each comprising a different version of a similar media player or entirely different media players. Accordingly, each instance of media tester110may be configured to communicate with a particular media player120. Although not required, in some embodiments each media player120is communicatively coupled to at least one accessory input device such as a keyboard, mouse, game controller, or remote control. As will be discussed in greater detail, in some embodiments media tester110may utilize test hooks in such input devices in addition to or in lieu of test hooks in a base unit of media players120in order to simulate human interaction. Test database135and media file database130may be centralized or distributed among one or more memory devices internal or external to media crawler125and media tester110.

Although interconnectivity may vary greatly among embodiments of scalable automated empirical testing of media files on media players, in the embodiment shown inFIG. 1, media tester110is communicatively coupled to media players120by communication link136, media tester110is communicatively coupled to media file database130by communication link137, media file database130is communicatively coupled to media crawler125by communication link138, media crawler125is communicatively coupled to test database135by communication link139and test database135is communicatively coupled to media tester110by communication link140. Each communication link136-140may comprise any one or a combination of the well-known communication links such as an internal bus, localized bus, private network or public network (e.g., Internet). Communication links136-140may comprise a single network or a combination of networks. Similarly, the type of communication on communication links136-140may be any of the well-known wired or wireless protocols. Those of ordinary skill in the art will recognize many interconnectivity alternatives to the one shown inFIG. 1. For example, in some embodiments communication links138and140may comprise local bus connections, communication links137and139may not exist and, instead, media tester110and media crawler125may be communicatively coupled by a communication link (not shown). Further still, the components shown inFIG. 1may be rearranged in various embodiments of scalable automated empirical testing of media files on media players which, in turn, would rearrange or eliminate communication links136-140. For example, in some embodiments media crawler125may be merged with media tester110in a single computer system.

Media files used for testing playback on media players120may be stored in media file database130. Media files stored in media file database130may number in the tens, hundreds, thousands, tens of thousands or more. Media crawler125may catalog media files stored in media file database130in order to create a test wordlist. Media crawler125may run periodically or upon request to catalog new media files added to media file database130. Each work item in the wordlist may contain a link to a media file in media file database130. Each work item may also specify test parameters such as which parameters to monitor and an acceptable performance profile for one or more monitored parameters. Each work item may also further define testing such as by specifying repetitious playback on one or another specific number of media players120.

One or more instances of media tester110may access the wordlist to select an item of work. The ratio of instances of media tester110to media players120may be 1:1 or any other ratio. Therefore, multiple instances of media tester110may be simultaneously accessing a wordlist stored in test database135. In some embodiments, items of work may be checked out (e.g., registered) and/or randomly selected in order to avoid collisions with other instances of media tester110that may also be attempting to select work items. For tests requiring repetitive playback on one or more media players120, in some embodiments there may be one work item per playback in the wordlist while other embodiments may operate in a different manner. For example, in some embodiments, work items may specify playback testing details. In some embodiments, with a single instance or multiple instances of media tester110, media tester(s)110may interact with the number of media players120called for in the work item. In embodiments with multiple instances of media tester110that are able to communicate with one another, a work item requiring multiple media testers120may be handled by a first instance of media tester110queuing work items with other instances of media tester. In other embodiments, an instance of media tester110may generate multiple work items from a single work item, thereby updating a wordlist. In such a case, an instance of media tester110may assign a priority that other instances of media tester110are required to process in advance of other work items.

Media tester110may have multiple test modes. For example, a first mode may test a media file once while a second test mode may test a media file multiple times. In some embodiments, test modes may define the ratio of media testers110to media players120. In one test mode the ratio may be 1:1 while in other modes it may be different. In some embodiments test modes may be modified by work items, e.g., the ratio may change in response to a work item requiring repetitive play on multiple media players120. Multiple test modes is one way to permit dynamic scaling of the testing system. Each instance of media tester110may have multiple operating modes independent of other instances. In addition to or in lieu of specification in each work item, each operating mode of each instance of media tester110may specify test parameters and their acceptable performance profiles or further define testing such as by specifying repetitious playback on one or more media players. As further examples, in one test mode media tester110may randomly select work items while in others it may check out work items. In some embodiments media tester110may register with test database135when is starts testing a media file. Thus, test modes may be used for many reasons including, but not limited to, avoiding collisions, maximizing processing of work items, avoiding saturation of instances of media tester110, avoiding unnecessary duplicative testing, etc.

While processing a work item in preparation for testing, an instance of media tester110may acquire and download the associated media file to one or more media players120in accordance with test directives specified in the work item, an operating mode, or elsewhere. One reason for repetitive testing of a media file on one or more media players120may be to audit previously identified failures, which may provide further insight to the cause of failures, e.g., by determining whether a failure is intermittent. In some embodiments, downloads may be performed by simulating human interaction with media players120. For example, through test hooks in media player120, media tester110may communicate the information to media player120that would occur if a USB memory device or other media type had been inserted into media player120and a media file on the USB memory device had been selected to copy and play or simply to play from the USB memory device. Similarly, media tester110may communicate the information to media player120that would occur if a user had accessed XBOX LIVE, selected the media file and downloaded it.

Following download to one or more media players120, a media file may be played by media players120using simulated human interaction with media players120. The simulated human interaction may utilize test hooks in a base unit (e.g., game console) and/or accessory unit (e.g., game controller, remote control) of media players120. As such, simulated human interaction may be provided by media tester110to the base unit and/or accessory unit of media players120. Just as a person using media players120to play media files would be, and in some embodiments, media tester110is able to control media players120. In some embodiments, simulated human interaction with media players120to play media files may involve providing to media player120the commands and data that it would receive if a user had actually pressed a button or otherwise interacted with a user interface to media player120such as a game controller or remote control. For example, if a user is required to navigate a menu using an input device in order to play a media file then each step a user must take is simulated by providing commands and/or data to media player120. In some embodiments, it may be necessary to first determine the existing state of media player120in order to determine how to interact with media player120to reach a desired state of media player120. Once the existing and desired states are known, media tester110may provide information, i.e., commands and/or data, to media player120to navigate from the existing state to the desired state. Repetitious play may involve other variations of simulated human interaction in order to test aspects of media players120and/or media file that may not be tested by other simulated human interaction.

While a media file is being played by media player120, playback performance may be monitored, analyzed and reported in accordance with test directives specified in the work item, operating mode, or elsewhere. As well, in some embodiments, parameters and performance profiles may be fixed in media tester110. However defined, the specific parameters to be monitored and their acceptable performance profiles are used to define the scope of monitoring and to identify failures. Utilizing test hooks in media players120, media players120provide information about monitored parameters. The information is tested against performance profiles to identify failures. Some examples of failures may include: (a) dropped frames, such as when samples in a media file are being discarded because media player120cannot keep up; (b) graphics completion events, e.g., Direct 3D Swap or D3D Present, exceeding frame rate, e.g., 16.67 mS at 60 Hz; (c) CPU core saturation; (d) playback exceeds length of media file, which may indicate a playback slowdown, temporary lockup/soft hang or a permanent lock up/crash. Such failures may be listed in a test report.

Test reports reporting the results of playback performance monitoring may be stored, for example, in test database135. Test reports may be accompanied by instructions to reproduce failures. They may also cross-reference each failure to content in media files as well as the source code of the application running on media player120that played the media file when the failure occurred. Information about a failure may be reported based on whether a failure is known. New failures may be treated differently from known failures in terms of the amount of information provided in a report and whether and to what extent to any files should accompany the report such as a call stack, debug spew and/or dump file providing the contents of memory when the failure was encountered. Such additional files may be most useful to debug crashes.

Failures may be analyzed before and/or after generation of test reports. If analyzed prior to the generation of a report then analyses may be incorporated therein. Failures may be analyzed to categorize, rank, prioritize or assign them to a review team or otherwise render a judgment pertaining to the failures. Such analyses may also access additional information from test database135concerning, for example, previous test results in order to determine whether a failure is known or new. Combining previous test results with new test results may allow additional analyses. As previously noted, known failures may lead to less data logging. Additionally, as previously noted, failures may be audited by additional work items for follow-up testing. In some embodiments, failures may automatically generate additional work items for auditing. In some embodiments, the type of failure (e.g., its category, rank or priority) may dictate the details of automated follow-up testing such as the test parameters, performance profiles, number of tests and the number of media players involved in the follow-up testing. Such follow-up testing may help determine whether a failure is persistent, intermittent or an anomaly. Upon completing a test, an instance of media tester110may continue with such follow-up testing or may recontact a test database135to select a new work item.

Irrespective of the timing of a failure audit immediately after the test identifying the failure or at subsequent time, as where additional work item(s) is (are) automatically generated from the failure or manually generated upon review and routinely selected from the wordlist by media tester110, a failure audit may comprise running the same media file multiple times on one or more media players120using the same or different testing parameters, performance profiles and simulated human interaction relative to the original test. The auditing tests may be carried out in a similar procedure, e.g., providing the media file to one or more media players120, simulating human interaction with each media player120, playing the media file on each media player120, monitoring each media player120during playback of the media file, reporting playback performance, including any failures that occur, during playback of the media file on each media player120and perhaps analyzing and/or auditing any failures. To avoid using the same media player(s)120for auditing, media players120used to audit a failure may be randomly selected.

Test reports may be reviewed by one or more people or teams. In embodiments where a list of review teams and logic to assign failures to them are known, media tester110may assign failures to particular review teams. For example, review teams may be selected by failure priority, failure category, software build of media player120, and/or other facts. For example, a crash failure may be assigned directly to a development team without first involving a quality assurance team, which may avoid unnecessary delay and expense. Media tester110may be distributed to media player110and media file developers to see failures reproduced firsthand.

In addition to media files, scalable automated empirical testing of media files on media players applies equally well to media applications such as video games. However, media applications may require more extensive simulated human interaction with media players120to empirically test playback. Work items for media applications such as games requiring constant interaction to play may be accompanied by supporting files used during testing to simulate human interaction with an input device such as a game controller or remote control. Performance monitoring of media applications may be carried out in the same manner as with media files, although failure reports may cross-reference failures to source code in media applications.

FIG. 2illustrates exemplary configurations of and interface between a media player and media tester in accordance with an aspect of scalable automated empirical testing of media files on media players. As indicated inFIG. 2, numerous components have been simplified or omitted fromFIG. 2in order to focus on selected features of scalable automated empirical testing of media files on media players. For example, a controller portion of media tester210utilized to select work items and acquire media files is not shown in order to focus on exemplary communication between media tester210and media player220. Further, exemplary components inFIG. 2may be merged or otherwise rearranged in other embodiments of scalable automated empirical testing of media files on media players. In the embodiment shown inFIG. 2, exemplary test configuration200comprises media tester210and media tester220as generally described in the discussion ofFIG. 1. Influencing the design of various embodiments of scalable automated empirical testing of media files on media players is a primary objective of empirical testing to closely approximate actual use of media player220. Media tester210comprises user interface (“UI”) navigator212, performance monitor214and crash monitor216. Media player220comprises UI location test hook222, user input test hook223, performance test hook224and debugger226. UI navigator212is communicatively coupled to UI location test hook222by communication link241. UI navigator212is also communicatively coupled to at least one user input test hook223by communication link242. Performance monitor214is communicatively coupled to performance test hook224by communication link243. Crash monitor216is communicatively coupled to debugger226by communication link244.

Having downloaded a media file to media player220, media tester210begins by finding out the existing state of the menu system of media player220. This may be done, for example, through communication between UI navigator212and UI location test hook222, which will be discussed in greater detail with regard toFIG. 3. After the existing state of the menu system of media player220has been determined, UI navigator212can begin to move through the menu system to a desired state to play the media file, an example of which is shown inFIG. 4. This may be done, for example, through communication between UI navigator212and user input test hook223that simulates human interaction with at least one user input to media player220. As previously noted, there may be one or more user input test hooks223in the base unit (e.g., game console) or accessory (e.g., game controller, remote control) of media player220. User input test hooks223permit simulated human interaction with one or more user inputs. User inputs may comprise, for example, any one or more of the wide variety of ways that a human can interact with the base unit or accessory to media player220through human interface elements, such as buttons, pads, paddles, pedals and directional devices including, but not limited to, directional pads, joysticks and wheels.

During playback of the media file, performance monitor214monitors selected information provided through performance test hook224. Selected information provided through performance test hook224may include, for example, video frame processing, graphics completion events, CPU saturation and playback time. The information is tested against performance profiles to identify failures. Some examples of failures may include: (a) dropped frames, such as when samples in a media file are being discarded because media player120cannot keep up; (b) graphics completion events, e.g., Direct 3D Swap or D3D Present, exceeding frame rate, e.g., 16.67 mS at 60 Hz; (c) CPU core saturation; (d) playback exceeds length of media file, which may indicate a playback slowdown, temporary lockup/soft hang or a permanent lock up/crash. As previously discussed, such failures may be listed in a test report. Failures logged in the test report may include instructions on how to reproduce the failures accompanied by the locations in the media file where the failures were encountered during playback. Performance monitor214may also analyze failures, assign one or more review teams and instigate auditing of the failures, as previously discussed with regard toFIG. 1. Basic failures may first be processed by a quality assurance team to further rank or prioritize failures depending on their impact on customers. A development team may be assigned more serious priority failures without delay.

Crash monitor216watches for catastrophic type failures by monitoring information provided through debugger226. Upon a crash, in some embodiments crash monitor216may log all information logged by performance monitor214plus pertinent information useful to developers to diagnose and solve such catastrophic failures. In some embodiments, crash monitor216may consult test database135to determine whether a crash failure is known or not. If the crash failure is new then, in some embodiments, the full contents of memory in media player220are saved into a full dump file that can be available along with the test report on test database135. A developer may load the dump file into Visual Studio or a similar debugging tool to help diagnose the crash failure. Crash monitor216may also cross reference a crash failure the source code of the application running on media player220that was playing the media file. Crash monitor216may also log in the test report where the crash occurred in the media file along with the call stack of the crash and pertinent contents of debugger226. In situations where a crash failure is known, reduced failure logging may be performed by crash monitor216so long as the results can be audited and reproduced. Reduced logging for known failures may comprise saving a mini-dump file instead of a full dump file. Crash failures may be logged as code defect bugs and automatically assigned to a development team without quality assurance team review, which saves time and expense.

As shown, monitoring happens in the same end-to-end environment used by consumers rather than in a disjointed or separate test harness. An example of why this is a superior method of testing pertains to the interaction between media playback and the user interface of media player220. There are failure situations where performance monitor214and crash monitor216report success because a media file played back fine even though information was not properly communicated to the user interface causing media player220to lock up. In embodiments of scalable automated empirical testing of media files on media players, UI navigator212can accurately log this scenario as a failure.

FIG. 3illustrates an exemplary communication between media player and media tester in accordance with an aspect of scalable automated empirical testing of media files on media players. Unless always beginning from a known location, such as following a reset, in order to navigate the menu system in media player320, UI navigator312must identify the menu system its existing state in media player320. In order to determine the existing state of the menu system, UI navigator316may send a command to UI location test hook326requiring media player320to identify the existing state of the menu system. Command inquiry345may be formatted as shown inFIG. 3, “dash!WhereAmI.” The portion of the command “dash,” as in the XBOX graphical user interface known as Dashboard, identifies the menu system process or program. The portion of the command “WhereAmI” identifies the inquiry made to the menu system process or program. UI location test hook326issues a response346to command inquiry345, which is provided to UI navigator316. Response346may be formatted as in shownFIG. 3, “200-section://30000F08,gamesbla#gamesSignedOut.xur!navCreateProfile.” The portion of response346“200-section://30000F08,gamesbla#gamesSignedOut.xur” comprises a string identifying the active menu screen output by media player320to a display viewed by a user. The portion of response346, “navCreateProfile” identifies the user input (e.g., button) currently highlighted in the user interface (e.g., game controller, remote control). Although response346is shown as one string, a response by UI location test hook322may comprise one or more strings. UI navigator312compares response346to a map of the menu system of media player320in order to determine how to proceed to a desired state in the menu system to play the media file downloaded to media player320. UI navigator312navigates to a desired state by issuing commands to user input test hook223and issuing command inquiries to UI location test hook222to ensure that media player220responds appropriately to the simulated user inputs.

FIG. 4illustrates an exemplary navigation through a menu system in accordance with an aspect of scalable automated empirical testing of media files on media players. In the embodiment shown inFIG. 4, menu system400illustrates a portion of the XBOX 360 Dashboard, which is the graphical user interface of the XBOX 360. The Dashboard serves as the control center for the XBOX 360 video game and entertainment system. The Dashboard comprises five “blades” or tabs that can be selected by users to flip between menu screens. The five menu screens are XBOX LIVE, Games, Media, Marketplace (not shown) and System (not shown). If inactive, the Dashboard can be activated in various ways, including by selecting an XBOX Guide button on a game pad, also known as a D-pad (i.e. directional pad), or the left stick of a game controller. The XBOX LIVE Blade menu allows users to, among other things, enter XBOX LIVE. The Games Blade menu allows users to, among other things, access their collection of games. The Media Blade menu allows users to, among other things, download and play live and recorded music, videos, movies, television shows, view pictures and slideshows, and create play lists from many sources internal and external to the XBOX 360. The Marketplace Blade menu allows users to, among other things, download movies, television shows, trailers, demonstrations, short films, game themes and pictures. The System Blade menu allows users to, among other things, edit XBOX 360 console playback settings, manage a connection to a computer, network or XBOX LIVE, and manage memory such as an XBOX 360 hard drive.

The portion of the XBOX 360 Dashboard shown inFIG. 4illustrates an exemplary navigation path, i.e., from existing state dashboard not active program405through XBOX LIVE blade410, games blade415, media blade420, music button425, pictures button430and video button435to desired state video file selection screen440. Each bubble represents a state of menu system400. As shown inFIG. 4, the existing state is dashboard not active program405. With knowledge of menu system400and its existing state acquired as shown inFIG. 3, for example, UI navigator312may issue state to state transition commands to user input test hook223to navigate to the desired state, e.g., video file selection screen440. As shown inFIG. 4, UI navigator212issues seven transition commands450-456to navigate from dashboard not active program405to video file selection screen440. Transition commands may be extendable. Each transition450-456may simulate one or more simulated human inputs. For example, transition command450from dashboard not active program405to XBOX LIVE blade410may simulate a user pressing a sequence of buttons to bring up the system blade menu (not shown) and load the Dashboard program. Other transitions, e.g., transitions451-456, may simulate a user pressing the Right, Down or “A” buttons on an XBOX 360 game pad (not shown). In some embodiments, UI navigator212may inquire after each transition command450-456about the existing state of menu system400by, for example, issuing command inquiry345to UI location test hook322. In some embodiments, if response346indicates the existing state is as expected or further along in the path then navigation continues without logging any failure. If response346indicates the existing state is not on the expected path then UI navigator212may log a failure and derive a new path to the desired state. In some embodiments, following a failure, media tester210may increase monitoring or testing, which may mean more slowly traversing menu system400or navigating multiple paths through menu system400. Deriving new paths through menu system400permits media tester210to overcome obstacles such as the presence or absence of optional states, response to user button presses/holds, atomicity of menu navigation and connectivity issues with media player220.

UI Navigator212manipulates media player220just as a user is able to do. Where a media file being tested is a video, for example, UI navigator212navigates to a state to select and play the video, e.g., video file selection screen440. UI navigator312may issue other commands in accordance with test specifications. For example, in addition to issuing commands to play a media file, UI navigator312may also issue commands to pause, stop, forward, rewind, etc. to exercise all or selected portions of media file content and code paths in media player220.

Exemplary Consumer Devices and Platform for Media Tester and Media Crawler

Without limitation,FIG. 5is a block diagram of an exemplary game console in which various aspects of scalable automated empirical testing of media files on media players can be implemented. An exemplary consumer device comprises, for example, a game console, such as an XBOX 360 game console, desktop, laptop, handheld or other computer, cellular telephone or any other media playback device.FIG. 6provides a block diagram of an exemplary computing environment as a representative consumer device and as a suitable platform on which media tester and/or media crawler may operate.

FIG. 5is a block diagram of an exemplary consumer device that can playback media files. Consumer device, i.e., game console,500comprises hardware, firmware and software. Game console500executes game applications and plays media files (not shown). For purposes of simplicity, not all components or interconnectivity are shown and some components have been merged in exemplary game console500.

Game console500comprises central processing unit (CPU)501, which has level1(L1) cache502, level2(L2) cache504, and ROM (Read-Only Memory)506. Level1and Level2cache502,504temporarily store executable instructions and/or data, thereby improving processing speed and throughput. ROM506may store basic executable instructions (i.e. firmware) that are loaded during an initial phase of a boot process when the game console500is initially powered. Alternatively, the executable code that is loaded during the initial boot phase can be stored in another type of non-volatile memory. ROM506, or the alternative non-volatile memory need not be embedded within CPU501. Game console500may optionally be a multi-processor system. For example, game console500may have three processors501,503, and505, where processors503and505, which may be similar or dissimilar to processor501. CPU's501,503and503may share cache memory (not shown).

Game console500further comprises graphics processing unit (GPU)508, which is coupled to CPU501, and any additional processors such as CPUs503,505, by a bus. GPU508is also coupled by one or more busses each to memory controller510, I/O (input/output) hub518and video codec (coder/decoder)514. Memory controller510and video codec514may form part of GPU508. GPU508, in addition to video processing functionality, may comprise functionality commonly referred to as Northbridge. Northbridge functionality generally comprises a high speed memory and video hub having a memory controller and a video controller. In exemplary game console500, both CPU501and I/O hub (Southbridge)518access memory512through Northbridge functionality in GPU508. Memory controller510facilitates access to various types of memory512, which may be RAM (Random Access Memory) or other variety of memory.

GPU508and video codec514together form a video processing pipeline for high speed, high resolution graphics processing required by many game applications. Data is carried from GPU508to/from video codec514via a bi-directional bus. This video processing pipeline outputs data to A/V (audio/video) port540for transmission to a television or other video display device (not shown). Game console500may have its own integrated display. Not shown is a digital to analog converter (DAC) that may be coupled between video codec514and A/V port540.

Game console500further comprises I/O hub518, which may comprise, among other functionality, functionality commonly referred to as Southbridge. Southbridge functionality generally performs and controls functions that are relatively slow compared to functions performed and controlled by Northbridge. I/O hub518comprises I/O controller520, system management controller522, audio processing unit523, network interface controller524, USB host controllers526,528and front panel I/O subassembly530. USB controllers526,528serve as hosts for peripheral controllers542(1),542(2), wireless adapter548, and memory unit546(e.g., flash memory, CD/DVD ROM, hard drive, other removable media). Network interface524and/or wireless adapter548provide access to a network (e.g., local area network (LAN), Internet) and may be any of a wide variety of various wired or wireless interface components including an Ethernet card, modem, Bluetooth module, and the like.

System memory543may store application data that is loaded during the boot process. Media drive544may comprise, for example, a DVD/CD drive, hard drive or other fixed or removable media reader and/or writer. Game application data may be read from and/or written to media via media drive544for execution, playback, etc. by game console500. Media drive544is connected to I/O controller520via a bus, such as a Serial ATA bus or other high speed connection (e.g., IEEE 5394). Game console500may include hard disk552, which may be used, for example, to store an operating system, game applications, game data or other types of data.

System management controller522provides a variety of service functions for game console500. Audio processing unit523and audio codec532form a corresponding audio processing pipeline that may provide, for example, high fidelity, 5.1 Dolby Digital, 3D, and stereo audio processing of sounds produced by, for example, a game application. Audio data is carried between audio processing unit523and audio codec532via a communication link. The audio processing pipeline outputs audio data to A/V port540for implementation by a device having audio capabilities.

Front panel I/O subassembly530supports the functionality of various controls such as power button550and eject button552, as well as any LEDs (light emitting diodes) or other indicators exposed on the outer surface of game console500. System power supply module536provides power to components of game console500while fan538cools them.

CPU501, GPU508, memory controller510, and various other components within game console500are interconnected via one or more buses, including serial and parallel buses, a memory bus, a peripheral bus, and a processor or local bus using any of a variety of bus architectures. As previously noted, not all buses or other connections are shown inFIG. 4.

When game console500is powered on or rebooted, application data and/or instructions can be loaded from system memory543, media drive544, hard disc553or other memory into memory512and/or caches502,504and executed on the CPU501. The game application being executed may present a graphical user interface that provides a consistent user experience when navigating to different media types available on or to game console500. Instructions and/or data accessible via media drive544, system memory543, hard disk553or other memory may be launched, played or otherwise accessed from these various sources to provide additional functionality to game console500.

Game console500may be operated as a standalone system by connecting the system to a television or other display. As previously noted, game console500may have an integrated display. In this stand alone mode, game console500may allow one or more users to interact with the system, watch movies, listen to music, play games and the like. Network interface524or wireless adapter548may allow game console500to be operated as a participant in a local or wide area network community.

FIG. 6is a block diagram of an exemplary computing environment in which various aspects of scalable automated empirical testing of media files on media players can be implemented. For example, the exemplary computing environment shown inFIG. 6illustrates and exemplary media player as well as a suitable computing environment in which scalable automated empirical testing of media files on media players or portion of it such as media tester program and/or media crawler program can be implemented. Although not required, various aspects of media tester and media crawler can be described in the general context of computer executable instructions, such as program modules, being executed by a computer, such as a personal computer, client workstation or a server. Generally, program modules include routines, programs, objects, components, data structures and the like that perform particular tasks or implement particular abstract data types. Moreover, implementation of media tester and media crawler can be practiced with other computer system configurations, including hand held devices, multi processor systems, microprocessor based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Further, media tester and media crawler also can be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

A computer system can be roughly divided into three component groups: the hardware component, the hardware/software interface system component, and the application programs component (also referred to as the “user component” or “software component”). In various embodiments of a computer system the hardware component may comprise central processing unit (CPU)621, memory (both ROM664and RAM625), basic input/output system (BIOS)666, and various input/output (I/O) devices such as keyboard640, mouse662, display645, and/or printer (not shown), among other components. The hardware component comprises the basic physical infrastructure for the computer system.

The application programs component comprises various software programs including but not limited to compilers, database systems, word processors, business programs, video games, and so forth. Application programs provide the means by which computer resources are utilized to solve problems, provide solutions, and process data for various users (machines, other computer systems, and/or end-users). In an example embodiment, application programs perform the functions associated with media tester and media crawler as described above.

The hardware/software interface system component comprises (and, in some embodiments, may solely consist of) an operating system that itself comprises, in most cases, a shell and a kernel. An “operating system” (OS) is a special program that acts as an intermediary between application programs and computer hardware. The hardware/software interface system component may also comprise a virtual machine manager (VMM), a Common Language Runtime (CLR) or its functional equivalent, a Java Virtual Machine (JVM) or its functional equivalent, or other such software components in the place of or in addition to the operating system in a computer system. A purpose of a hardware/software interface system is to provide an environment in which a user can execute application programs.

The hardware/software interface system is generally loaded into a computer system at startup and thereafter manages all of the application programs in the computer system. The application programs interact with the hardware/software interface system by requesting services via an application program interface (API). Some application programs enable end-users to interact with the hardware/software interface system via a user interface such as a command language or a graphical user interface (GUI).

A hardware/software interface system traditionally performs a variety of services for applications. In a multitasking hardware/software interface system where multiple programs may be running at the same time, the hardware/software interface system determines which applications should run in what order and how much time should be allowed for each application before switching to another application for a turn. The hardware/software interface system also manages the sharing of internal memory among multiple applications, and handles input and output to and from attached hardware devices such as hard disks, printers, and dial-up ports. The hardware/software interface system also sends messages to each application (and, in certain case, to the end-user) regarding the status of operations and any errors that may have occurred. The hardware/software interface system can also offload the management of batch jobs (e.g., printing) so that the initiating application is freed from this work and can resume other processing and/or operations. On computers that can provide parallel processing, a hardware/software interface system also manages dividing a program so that it runs on more than one processor at a time.

A hardware/software interface system shell (referred to as a “shell”) is an interactive end-user interface to a hardware/software interface system. (A shell may also be referred to as a “command interpreter” or, in an operating system, as an “operating system shell”). A shell is the outer layer of a hardware/software interface system that is directly accessible by application programs and/or end-users. In contrast to a shell, a kernel is a hardware/software interface system's innermost layer that interacts directly with the hardware components.

As shown inFIG. 6, an exemplary general purpose computing system in which scalable automated empirical testing of media files on media players may be implemented includes a conventional computing device660or the like, including a processing unit621, a system memory662, and a system bus623that couples various system components including the system memory to the processing unit621. Processing unit621may comprise, for example, a CPU, Northbridge and Southbridge chipset, among other components. The system bus623may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes read only memory (ROM)664and random access memory (RAM)625. A basic input/output system666(BIOS), containing basic routines that help to transfer information between elements within the computing device660, such as during start up, is stored in ROM664. The computing device660may further include a hard disk drive625for reading from and writing to a hard disk (not shown), a magnetic disk drive628(e.g., floppy drive) for reading from or writing to a removable magnetic disk629(e.g., floppy disk), and an optical disk drive630for reading from or writing to a removable optical disk631such as a CD ROM or other optical media. The hard disk drive625, magnetic disk drive628, and optical disk drive630are connected to the system bus623by a hard disk drive interface632, a magnetic disk drive interface633, and an optical drive interface634, respectively. The drives and their associated computer readable media provide non volatile storage of computer executable instructions, data structures, program modules and other data for the computing device660. Although the exemplary environment described herein employs a hard disk, a removable magnetic disk629, and a removable optical disk631, it should be appreciated by those skilled in the art that other types of computer readable media which can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like may also be used in the exemplary operating environment. Likewise, the exemplary environment may also include many types of monitoring devices such as heat sensors and security or fire alarm systems, and other sources of information.

A number of program modules comprising computer-executable instructions can be stored on any one or more computer-readable mediums such as the hard disk, magnetic disk629, optical disk631, ROM664, or RAM625, including an operating system635, one or more application programs636, other program modules635, and program data638. A user may enter commands and information into the computing device660through input devices such as a keyboard640and pointing device662(e.g., mouse). Other input devices (not shown) may include a microphone, joystick, game pad, satellite disk, scanner, or the like. These and other input devices are often connected to the processing unit621through a serial port interface646that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port, or universal serial bus (USB). Aside from mouse642, keyboard640and modem654, serial port I/F646may also be connected to a wireless communications device (not shown). A display645or other type of display device is also connected to the system bus623via an interface, such as a video adapter648. In addition to display645, computing devices typically include other peripheral output devices (not shown), such as speakers and printers. The exemplary environment ofFIG. 6also includes a host adapter655, Small Computer System Interface (SCSI) bus656, and an external storage device662connected to the SCSI bus656.

The computing device660may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer649. The remote computer649may be another computing device (e.g., personal computer), a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above relative to the computing device660, although only a memory storage device650(floppy drive) has been illustrated inFIG. 6. The logical connections depicted inFIG. 6include a local area network (LAN)651and a wide area network (WAN)652. Such networking environments are commonplace in offices, enterprise wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the computing device660is connected to the LAN651through a network interface or adapter653. When used in a WAN networking environment, the computing device660can include a modem654or other means for establishing communications over the wide area network652, such as the Internet. The modem654, which may be internal or external, is connected to the system bus623via the serial port interface646. In a networked environment, program modules depicted relative to the computing device660, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

While it is envisioned that numerous embodiments of scalable automated empirical testing of media files on media players are particularly well-suited for computerized systems, nothing in this document is intended to limit scalable automated empirical testing of media files on media players to such embodiments. On the contrary, as used herein the term “computer system” is intended to encompass any and all devices capable of storing and processing information and/or capable of using the stored information to control the behavior or execution of the device itself, regardless of whether such devices are electronic, mechanical, logical, or virtual in nature.

The functionality described herein can be implemented in connection with hardware or software or, where appropriate, with a combination of both. Thus, the methods, apparatuses and systems for scalable automated empirical testing of media files on media players, or certain aspects or portions thereof such as media tester110and media crawler125, can take the form of program code (i.e., instructions) and/or data embodied in tangible computer readable media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program code or data is loaded into and executed or read by a machine, such as a computer, the machine becomes an apparatus for implementing scalable automated empirical testing of media files on media players.

The program(s) can be implemented in assembly or machine language, if desired. In any case, the language can be a compiled or interpreted language, and combined with hardware implementations. The methods and apparatuses for implementing scalable automated empirical testing of media files on media players also can be practiced via communications embodied in the form of program code that is transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via any other form of transmission, wherein, when the program code is received and loaded into and executed by a machine, such as an EPROM, a gate array, a programmable logic device (PLD), a client computer, or the like. When implemented on a general-purpose processor, the program code combines with the processor to provide a unique apparatus that operates to invoke the functionality of media tester and media crawler. Additionally, any storage techniques used in connection with media tester and media crawler can invariably be a combination of hardware and software.

Empirical testing offers very useful real-world feedback because testing closely follows scenarios that customers follow while playing media files or media applications. Automated empirical testing navigates the same user interface, triggers the same code paths on the same hardware and results in the same memory footprints experienced by media players used by customers. As such, empirical test results are more realistic and useful than emulation test results. A scalable empirical testing system with ones, tens, hundreds or more media players operating 24 hours per day 365 days per year permits content providers to process large volumes of media files in short periods of time, which permits greater flexibility in making software improvements to media players and offers greater quality assurance in media files. A scalable empirical testing system permits testing throughout development and production of hardware and software builds of media players. A scalable empirical testing system also removes the burden of testing from customers, increases the level of confidence customers have in downloaded content and, ultimately, improves the quality reputation of content service providers having such a system.

While scalable automated empirical testing of media files on media players has been described in connection with the example embodiments of the various figures, it is to be understood that other similar embodiments can be used or modifications and additions can be made to the described embodiments for performing the same functions of scalable automated empirical testing of media files on media players without deviating there from. Therefore, scalable automated empirical testing of media files on media players as described herein should not be limited to any single embodiment, but rather should be construed in breadth and scope in accordance with the appended claims.