Automated measurement of mobile device application performance

Methods, systems and apparatuses, including computer program products, are described for determining a performance metric of a mobile computing device application. A test computing device captures a plurality of images displayed on a mobile computing device based on execution of a mobile computing device application. The test computing device determines a first property for first and second images of the plurality of images. A first performance parameter is set based on a difference between the first properties of the first and second images. The test computing device determines a first property for third and fourth images of the plurality of images. A second performance parameter is set based on a difference between the first properties of the third and fourth image. A performance metric is determined based on a difference between the first performance parameter and the second performance parameter.

TECHNICAL FIELD

This application relates generally to systems, methods and apparatuses, including computer program products, for automated quality assurance testing of mobile computing device applications, and determining a performance metric of a mobile computing device application.

BACKGROUND

Many mobile computing devices, such as tablets and smartphones, include one or more sensors for detecting when the orientation of the mobile computing device changes (e.g., from a landscape to a portrait orientation). In response to output from these sensors indicating an orientation change, mobile computing device applications may rotate the image displayed on the mobile computing device to match the orientation of the housing in order to provide a fluid, enjoyable user experience.

Some applications may also modify elements of the displayed image according to its current orientation. Rotating the displayed image can be processor intensive due to the calculations that must be carried out for resizing, repositioning, and re-rendering the various display elements. These intensive calculations can cause a delay that disrupts or hinders the fluidity of the user experience. As a result, developers of mobile computing device applications must exhaustively test their applications to verify performance in the event an orientation change occurs. However, existing quality assurance test systems and methods are incapable of testing a mobile computing device in multiple orientations, or require a large amount of human interaction to manually operate a test jig for rotating the mobile computing device between orientations. Further, current quality assurance test systems and methods require a test operator to manually review video or images of the display to determine mobile computing device application performance during an orientation change. Accordingly, the current process is time-consuming and prone to human error.

SUMMARY

Accordingly, there is a need for improved systems, methods and apparatuses for automated quality assurance testing of mobile computing device applications, and determining a performance metric of a mobile computing device application.

The invention, in one aspect, features a computerized method for determining a performance metric of a mobile computing device application. A test computing device captures a plurality of images displayed on a mobile computing device based on execution of a mobile computing device application. The test computing device determines a first property of a first image of the plurality of images and a first property of a second image of the plurality of images. The test computing device sets a first performance parameter based on a difference between the first property of the first image and the first property of the second image. The test computing device determines a first property of a third image of the plurality of images and a first property of a fourth image of the plurality of images. The test computing device sets a second performance parameter based on a difference between the first property of the third image and the first property of the fourth image. The test computing device determines a performance metric based on a difference between the first performance parameter and the second performance parameter.

The invention, in another aspect, features an automated test system for automated quality assurance testing of a mobile computing device application. The system includes a test computing device including a capture device, and a moveable support member in communication with the test computing device. The system further includes a mobile computing device having a display. The mobile computing device is coupled to the moveable support member in a first orientation. The moveable support member is capable of rotating the mobile computing device between the first orientation and a second orientation. The test computing device is programmed to capture a plurality of images displayed on a mobile computing device based on execution of a mobile computing device application. The test computing device is further programmed to determine a first property of a first image of the plurality of images and a first property of a second image of the plurality of images. The test computing device is further programmed to set a first performance parameter based on a difference between the first property of the first image and the first property of the second image. The test computing device is further programmed to determine a first property of a third image of the plurality of images and a first property of a fourth image of the plurality of images. The test computing device is further programmed to set a second performance parameter based on a difference between the first property of the third image and the first property of the fourth image. The test computing device is further programmed to determine a performance metric based on a difference between the first performance parameter and the second performance parameter.

The invention, in another aspect, features a computer program product, tangibly embodied in a non-transitory computer readable storage device, for determining a performance metric of a mobile computing device application. The computer program product includes instructions operable to cause a test computing device to capture a plurality of images displayed on a mobile computing device based on execution of a mobile computing device application. The computer program product includes instructions operable to cause the test computing device to determine a first property of a first image of the plurality of images and a first property of a second image of the plurality of images. The computer program product includes instructions operable to cause the test computing device to set a first performance parameter based on a difference between the first property of the first image and the first property of the second image. The computer program product includes instructions operable to cause the test computing device to determine a first property of a third image of the plurality of images and a first property of a fourth image of the plurality of images. The computer program product includes instructions operable to cause the test computing device to set a second performance parameter based on a difference between the first property of the third image and the first property of the fourth image. The computer program product includes instructions operable to cause the test computing device to determine a performance metric based on a difference between the first performance parameter and the second performance parameter.

Any of the above aspects can include one or more of the following features. In some embodiments, determining the first property of the first image includes computing a first luminance value based on a plurality of pixels of the first image, and determining the first property of the second image includes a second luminance value based on a plurality of pixels of the second image. In some embodiments, setting the first performance parameter based on the difference between the first property of the first image and the first property of the second image includes determining, by the test computing device, the difference between the first property of the first image and the first property of the second image exceeds a first predetermined threshold, and setting, by the test computing device, the first performance parameter based on a time the first image was displayed on the mobile computing device.

In some embodiments determining the first property of the third image includes computing a third luminance value based on a plurality of pixels of the third image, and determining the first property of the fourth image includes computing a fourth luminance value based on a plurality of pixels of the fourth image. In some embodiments, setting the second performance parameter based on the difference between the first property of the third image and the first property of the fourth image includes determining, by the test computing device, the first property of the third image and the first property of the fourth image are substantially equal, and setting, by the test computing device, the second performance parameter based on a time the third image was displayed on the mobile computing device.

In some embodiments, determining the first property of the first image and the first property of the second image includes: computing, by the test computing device, a pixel luminance value for each pixel of the first image; creating, by the test computing device, a first set of pixels including pixels of the first image having a predetermined range of pixel luminance values, where the first set of pixels comprises a predetermined number of pixels of the first image; computing, by the test computing device, a first luminance value based on a mean of the pixel luminance values of the pixels of the first set of pixels; creating, by the test computing device, a second set of pixels comprising pixels of the second image, where each pixel of the second set of pixels has a position corresponding to a position of a pixel of the first set of pixels; computing, by the test computing device, a pixel luminance value for each pixel of the second set of pixels; and computing, by the test computing device, a second luminance value based on a mean of the pixel luminance values of the pixels of the second set of pixels.

In some embodiments, determining the first property of the third image of the plurality of images and the first property of the fourth image of the plurality of images includes: creating, by the test computing device, a third set of pixels including pixels of the third image, wherein each pixel of the third set of pixels has a position corresponding to a position of a pixel of the first set of pixels; computing, by the test computing device, a pixel luminance value for each pixel of the third set of pixels; computing, by the test computing device, a third luminance value based on a mean of the pixel luminance values of the pixels of the third set of pixels; creating, by the test computing device, a fourth set of pixels comprising pixels of the fourth image, where each pixel of the fourth set of pixels has a position corresponding to a position of a pixel of the first set of pixels; computing, by the test computing device, a pixel luminance value for each pixel of the fourth set of pixels; and computing, by the test computing device, a fourth luminance value based on a mean of the pixel luminance values of the pixels of the fourth set of pixel.

In some embodiments, the third image and the fourth image are captured consecutively in time by the test computing device. In some embodiments, the first set of pixels includes between fifteen and thirty percent of a total number of pixels of the first image.

In some embodiments, the plurality of images displayed on the mobile computing device includes a video of an image displayed the mobile computing device changing from a first orientation to a second orientation during the first predetermined period of time.

In some embodiments, capturing the plurality of images displayed on the mobile computing device further includes capturing, by test computing device, a video of a reflection of the mobile computing device on a reflective element for a predetermined period of time, and extracting, by the test computing device, the plurality of images from the captured video. The plurality of images shows the reflection of the mobile computing device display at predetermined intervals during the predetermined period of time. In some embodiments, the plurality of images are extracted from the captured video at a rate of substantially sixty images per second.

In some embodiments, the performance metric comprises a duration of time for an image displayed on the mobile computing device to change from the first orientation to the second orientation. In some embodiments, the pixel luminance for each pixel of the first image is based on a weighted sum of a plurality of pixel color component intensity values of each pixel.

In some embodiments, the plurality of images displayed on the mobile computing device are captured in response to a first command transmitted by the mobile computing device executing a test script received from a remote computing device. The mobile computing device changes from a first orientation to a second orientation based on a second command transmitted to a moveable support member coupled to the mobile computing device.

DETAILED DESCRIPTION

FIG. 1is a block diagram of a system100in accordance with embodiments of the invention described herein. System100includes mobile computing device105coupled to arm115. Arm115can be a moveable support member (e.g., automated mechanical arm, robotic arm) having one or more joints capable of motion (e.g., rotational motion, linear motion). In some embodiments, arm115is capable of positioning mobile computing device105in a first orientation (e.g., landscape orientation), and rotating mobile computing device105substantially ninety degrees about an axis of rotation such that mobile computing device105is positioned in a second orientation (e.g., portrait orientation). In some examples, arm115can be operated to alternate the position of mobile computing device105between the first and second orientations. In some examples, arm115can rotate mobile computing device105substantially 360 degrees about an axis of rotation.

Mobile computing device105can be coupled to arm115via a gripper or holder having features to engage the housing of mobile computing device105. For example, mobile computing device105can be coupled to arm115using one or more of an adjustable vise, a spring clamp, a hydraulic clamp, a magnetic mount, a suction cup, and a vacuum.

Examples of mobile computing device105include tablet computers, smartphones, and other mobile computing devices known in the art.

FIG. 2is a block diagram200of an exemplary embodiment of mobile computing device105. Mobile computing device105includes processing subsystem205in communication with sensor interface210for accessing a variety of sensors215. Processing subsystem205generally includes a processor, volatile and non-volatile memory, and other logic for managing and interfacing with other components of mobile computing device105. Processing subsystem205is programmed with computer software instructions enabling it to perform computations and operations for executing mobile computing device applications and test scripts as described herein in conjunction with the other components of mobile computing device105. Sensor interface210includes circuitry to facilitate access to sensors215. In some embodiments, sensor interface210includes a co-processor in communication with one or more of sensors215for collecting and processing sensor data from sensors215.

Sensors215can include a plurality of sensors for detecting and/or measuring properties of mobile computing device105and its location, and providing corresponding output (e.g., electrical signal, optical signal). For example, sensors215can include sensors responsive to changes in the altitude of mobile computing device105(e.g., barometric pressure sensor, pressure altimeter, barometric altimeter). Sensors215can also include sensors responsive to a change in moisture content in the atmosphere (e.g., hygrometer), and sensors responsive to changes in the strength of a magnetic field (e.g., magnetometer, teslameter, gaussmeter). Sensors215can further include sensors responsive to changes in properties of sound (e.g., microphone, sound transducer) such as changes in the amplitude, tone, pitch, and/or duration of sound. In some examples, sensors215include sensors responsive to changes in properties of light (e.g. ambient light sensor, photodetector, phototransistor) such as changes in intensity and wavelength.

In some embodiments, sensors215includes sensors responsive to a change in linear acceleration (e.g., accelerometer) and angular velocity (e.g., gyroscope), and the output from the sensors comprises signals or data corresponding to a single axis or a plurality of axes (e.g., 2-axis sensor, 3-axis sensor).

Sensors215can further include other sensors such as an image sensor (e.g., camera), temperature sensor (e.g., thermometer), and biometric sensor (e.g., fingerprint reader). In some embodiments, mobile computing device105includes a plurality of a given sensor type for making multi-channel “diversity” measurements of certain properties.

Processing subsystem205is further in communication with several other components and subsystems of mobile computing device105such as communications220, display225, I/O230, light source235, speaker240, and power245. Although the example inFIG. 2shows a shared connection between processing subsystem205and all of the other components and subsystems of mobile computing device105, it should be understood that any number of connections or interfaces can be used to connect processing subsystem205to the other components and subsystems of mobile computing device105.

Communications220includes circuitry to facilitate communications between mobile computing device105and other computing and peripheral devices using various protocols over various transmission mediums. For example, communications220can include circuitry for communicating using wireless technology (e.g., Wi-Fi, cellular, GPS, Bluetooth) and wired technology (e.g., wired Ethernet, USB, optical, Lightning, FireWire). Display225can include a flat panel display or screen (e.g., LCD display, OLED display) and associated circuitry for controlling the display of images on the display. In some embodiments, display225further includes a touch panel or touchscreen display as an input device to mobile computing device105.

I/O230can include circuitry corresponding to one or more components for providing input to mobile computing device105(e.g., button, slider, switch), and providing output or feedback from mobile computing device105(e.g., audio signal output, video signal output, vibration circuit). Light source235can include one or more solid-state light sources (e.g., LED, OLED, PLED). In some embodiments, light source235includes components for enhancing or directing light from the solid state light source (e.g., reflector, lens, diffuser, light guide, light pipe) and/or circuitry for controlling the operation of light source235. Speaker240can include one or more sound-producing components (e.g., speakers) and related components for enhancing or directing sound (e.g., resonance chamber, amplifier). Power245can include a battery for providing power to the various components of mobile computing device105. Power245can also include circuitry for accepting power provided from a wired power source (e.g., AC/DC adapter, USB port), and charger circuitry for charging the battery.

Returning toFIG. 1, system100further includes test computing device135and capture device110in optical communication with reflective element120via optical path125. Capture device110can include a camera or image sensor and corresponding circuitry for capturing video and/or images. In some embodiments, capture device110is a component of test computing device135. In some embodiments, capture device110and test computing device135are devices having a substantially similar architecture to mobile computing device105. In some embodiments, capture device110stores and processes captured video according to the methods described herein. In some embodiments, capture device110is a digital camera coupled to a test computing device that communicates captured digital videos and images to the test computing device via a wired or wireless communications protocol. In some embodiments, test computing device135is a standalone computing device in communication with the components of system100.

Reflective element120can be any element having a reflective surface capable of producing a reflected image of an object placed in front of it. In some examples, reflective element120is a mirror. In some examples reflective element120includes a glass or other planar substrate coated with one or more metals and/or paint. In some embodiments, reflective element120is shaped as a square having sides at least as long as the longest side of mobile computing device105.

As shown inFIG. 1, reflective element120can be positioned in front of mobile computing device105such that a reflected image of mobile computing device105is formed by reflective element120. For example, mobile computing device105can be coupled to arm115in a landscape orientation with its display facing reflective element120, and substantially parallel to it. In some embodiments, mobile computing device105can be coupled to arm115with its display facing reflective element120at an acute angle.

Capture device110can be positioned adjacent to or behind and offset from mobile computing device105such that an image sensor of capture device110is in optical communication with the reflected image of mobile computing device105in reflective element120. In the example shown inFIG. 1, capture device110is positioned behind and above mobile computing device105, creating optical path125between the image sensor of capture device110and the image of mobile computing device105reflected in reflective element120. However, it should be understood that capture device110can be positioned at other locations with respect to mobile computing device105without departing from the scope of the technology. In some embodiments, capture device110and/or test computing device135are coupled to an automated mechanical arm substantially similar to arm115. In some embodiments, capture device110and/or test computing device135are coupled to a stationary mechanical arm or bracket.

In some embodiments, the components of system100described above are mounted within a housing or enclosure.

In various embodiments of the invention, the components of system100can communicate using one or more wired or wireless communications protocols. (Communications between the components of system100are represented generally by communications paths130.) For example, mobile computing device105, test computing device135, capture device110and arm115can be connected to a local network, such as a LAN, or a wide area network (“WAN”), such as the Internet and/or a cellular network, or alternatively, a network comprising components of both a LAN and a WAN, or any other type of network known in the art. In some embodiments, two or more components of system100are in communication using near field communications. In some examples, one or more of mobile computing device105, test computing device135, and capture device110receive power from and communicate over a cable connected to a wired interface connector (e.g., lightning, USB OTG, USB Type-C, PDMI, custom/proprietary dock connector).

The components of system100can further be in network communication with a remote computing device (not shown), and the remote computing device can perform various operations related to the methods described herein. In some embodiments, the remote computing device can transmit test scripts to mobile computing device105and commands to cause mobile computing device105to execute a test script. In some embodiments, the remote computing device can further control test computing device135, capture device110, and/or arm115using commands that are communicated over a network.

In some examples, mobile device105can communicate directly with arm115over a wired or wireless interface. For example, mobile device105can send commands to arm115to cause arm115to perform a movement (e.g., rotate mobile device105about an axis of rotation) to change the orientation of mobile device105. In some embodiments, mobile device105transmits commands for arm115to a computing device connected to arm115. In some examples, capture device110or test computing device135can control arm115in a substantially similar manner as described above with respect to mobile computing device105. In some embodiments, mobile computing device105, test computing device135, and capture device110can communicate directly or over a network for performing various steps of the methods described herein.

As shown inFIG. 1, mobile computing device105can display an image in landscape orientation when mobile computing device105is positioned in a landscape orientation. In some embodiments, the image shown on the display of mobile computing device105can be presented based on the execution of an application on mobile computing device105. In some embodiments, the application running on mobile computing device105can use output from one or more of sensors215(e.g., gyroscope, accelerometer) to determine that mobile computing device105device has been rotated from landscape to portrait orientation, and rotate the displayed image accordingly to also be in portrait orientation.

FIG. 3is a block diagram300showing system100in accordance with embodiments of the invention described herein. As shown inFIG. 3, arm115has rotated mobile computing device105substantially ninety degrees to a second orientation (e.g., portrait orientation), and the image displayed on mobile computing device105has also been rotated from landscape to portrait orientation.

Although the image shown in the examples discussed herein is a picture or photograph, it should be understood that mobile computing device105can display any number of image types based on images generated and presented by an application executing on mobile computing device105. In some examples, mobile computing device105displays a graphical user interface, and in response to a change in the orientation of mobile computing device105, one or more elements of the graphical user interface change. In some embodiments, the images presented by an application in portrait and landscape orientation include elements that differ by one or more of font size, font style, field length, number of input fields displayed, color, and shape.

As described above, mobile computing device105can execute a mobile computing device application that rotates the image displayed on mobile computing device105from landscape to portrait orientation, or vice versa, in response to mobile computing device105itself being rotated. However, for mobile computing device applications that also modify elements of the displayed image according to orientation, rotating the displayed image can be processor intensive due to the calculations that must be carried out for resizing, repositioning, and re-rendering the various display elements. This can cause a delay in the operation of the mobile computing device application that negatively impacts user experience. The speed with which a mobile computing device can rotate and update images displayed on its screen can be used as a metric for gauging performance of mobile computing device applications. Accordingly, systems, methods and apparatuses described herein can be used for automated quality assurance testing of a mobile computing device application, and in particular, for determining a performance metric of a mobile computing device application.

FIG. 4is a flow diagram of a method400for determining a performance metric of a mobile computing device application, using the system100shown inFIG. 1andFIG. 3. Mobile computing device105can be mounted to arm115in a first orientation (e.g., landscape orientation), with its display facing reflective element120. In some embodiments, mobile computing device105can execute a mobile computing device application and test script that have been preloaded on mobile computing device105. In some embodiments, a remote computing device can load a mobile computing device application and/or test script to mobile computing device105and cause mobile computing device105to begin executing the application and test script.

Test computing device135can capture (405) a plurality of images displayed on a mobile computing device based on execution of a mobile computing device application. Upon execution of a test script loaded as described above, mobile computing device105can transmit a command to test computing device135causing capture device110to begin capturing a plurality of images (e.g., video) of the reflection of mobile computing device105in reflective element120.

Further, in response to execution of the test script, mobile computing device105can transmit a command to arm115causing arm115to rotate mobile computing device105substantially ninety degrees about an axis of rotation such that mobile computing device105is positioned in a second orientation (e.g., portrait orientation).

Capture device110continues to capture video of the reflection of mobile computing device105in reflective element120throughout and after the time that arm115rotates mobile computing device105and the mobile computing device application executing on mobile computing device105rotates the image displayed on its screen. In some examples, capture device110captures video for a predetermined period of time (e.g., 2 seconds, 5 seconds, 10 seconds). In some examples, capture device110captures video until mobile computing device105transmits a command to computing device135causing capture device110to stop capturing video.

The captured video can be stored in a memory device local to capture device110or test computing device135. In some embodiments, the captured video is stored on a remote computing device or network attached storage device in network communication with capture device110and test computing device135.

The captured video of the reflection of mobile computing device105can be processed to extract a plurality of still images or frames of the images displayed on mobile computing device105at predetermined intervals during the capture period of time described above. For example, test computing device135can extract a predetermined number of images from the captured video for each second of captured video (e.g., 20 frames per second, 40 frames per second, 60 frames per second, 80 frames per second). In an exemplary embodiment, the captured video is five seconds in length, and test computing device135extracts images from the captured video at a rate of 60 frames per second, i.e., one image is extracted from the captured video substantially every 0.0167 seconds. Accordingly, in this example, 300 images of the reflection of mobile computing device105in reflective element120are extracted from five seconds of captured video.

Test computing device135can further process each extracted image to isolate the image displayed on the display of mobile computing device105. For example, image processing software and algorithms can be used to identify the edges of the mobile computing device105display and extract only the displayed image. This process can be facilitated by virtue of the captured images showing a reflection of mobile computing device105in reflective element120. In one aspect, the size and shape of reflective element120can aid in the capture of the images. For example, reflective element120can be a square having sides substantially as long as the longest side of the mobile computing device105housing. Accordingly, the reflection of mobile computing device105in reflective element120can show a substantially complete view of mobile computing device105including its entire housing whether mobile computing device105is in landscape or portrait orientation, or in transition between the two orientations.

In another aspect, the captured reflection of mobile computing device105in reflective element120further facilitates extraction of the images displayed on the mobile computing device105display. For example, in the reflected image of mobile computing device105, image processing software can readily differentiate between the mobile computing device105housing and the pixels of its backlit display. Accordingly, the edges of the mobile computing device105housing surrounding the display can be used as markers to delimit the boundaries of the display, and an image comprising just the display can be cropped and extracted from each of the images that were first extracted from the captured video of the reflection of mobile computing device105in reflective element120. In some embodiments, software such as Office Lens from Microsoft can be used to process the extracted images. However, it should be understood that other algorithms and techniques known in the art for edge detection and image processing can be used to process the extracted images without departing from the spirit of the invention.

The plurality of extracted images showing mobile computing device105transitioning from a first orientation to a second orientation can be tilted or angled with respect to one another. Further, the images of mobile computing device105in the first orientation can have a different aspect ratio from the images of mobile computing device105in the second orientation. Accordingly, the plurality of extracted images can be processed to normalize the orientation of each image and facilitate comparison between them. In some examples, image processing algorithms and techniques can be used to remap the position of the pixels of the extracted images such that there is substantially a 1:1 correspondence between the positions of the pixels of each extracted image. As an example, the plurality of extracted images can be normalized such that each image is shown in a landscape orientation, and a pixel located in the upper left hand corner of a first image has the same position (e.g., x,y coordinate) as a pixel located in the upper left hand corner of a second image.

After extracting and processing the plurality of images, test computing device135can perform various operations to determine when the mobile computing device application begins rotating the image displayed on the mobile computing device display in response to arm115rotating mobile computing device105from the first orientation to the second orientation. Test computing device135can determine (410) a first property of a first image of the plurality of images and a first property of a second image of the plurality of images. In some examples, the first property of the first and second images is a luminance value. Continuing the example described above in which 300 images are extracted from five seconds of captured video, test computing device135can compute a luminance value (e.g., brightness value) for the first of the 300 images based on a plurality of the pixels of the first image. In some examples, test computing device135computes the luminance of each pixel of the first image and sets the luminance value of the first image based on the mean of all the luminance values calculated for each pixel. A luminance value for a second image that occurred later in time than the first image can be similarly computed.

In some embodiments, the luminance of each pixel can be based on a weighted sum of a plurality of pixel color component intensity values of each pixel. For example, the luminance of each pixel can be determined by multiplying the red, green, and blue components of each pixel by corresponding constants and summing the result. For example, luminance can be determined based on the equation L=0.2126*R+0.7152*G+0.0722*B, where L is luminance, and R, G and B are numeric values corresponding to red, green and blue components of a particular pixel.

Based on a comparison of the luminance values calculated for the first image and the second image, test computing device135can determine when the image displayed on mobile computing device105began rotating. Test computing device135can set (415) a first performance parameter based on a difference between the first property of the first image and the first property of the second image. Test computing device135can compare the luminance values computed for the first and second images. Upon determining that the difference between the luminance values exceeds a predetermined threshold, test computing device135can set a first performance parameter indicating a time when the first image was displayed on mobile computing device105. In some embodiments, the first performance parameter is a numeric index indicating which image of the 300 captured images the first image is. In an exemplary embodiment, test computing device135sets the first performance parameter when the luminance values differ by substantially +/−20%. However, other thresholds for the difference between the luminance values of the first and second images can be used without departing from the spirit of the invention.

In some embodiments, the first image and the second image are images that were captured consecutively in time from the captured video (e.g., the first and second images of the 300 extracted images in the described example). In some embodiments, there can be a delay between the time when capture device110begins capturing video and the time when arm115begins rotating mobile computing device105, and the first image is not literally be the first image of the 300 extracted images. For example, if the difference between the luminance values of the first and second images does not exceed the predetermined threshold, test computing device135can instead compare the second image to the image that immediately followed it in time. In some embodiments, the first and second images are not consecutive in time. The method can continue in this manner until it is determined that the difference between two images has exceeded the predetermined threshold.

FIG. 5is a diagram500showing first image505and second image510in accordance with the invention. In the example shown inFIG. 5, first image505is the first of the 300 extracted images, and second image510is an image extracted a period of time after first image505when the image displayed on mobile computing device105was in the process of rotating from the first orientation to the second orientation. It can be understood that the luminance computed for first image505will be greater in magnitude than the luminance computed for second image510which includes substantially more black pixels than first image505. Therefore, the difference between the luminance values of first image505and second image510exceeds a predetermined threshold (e.g., +/−20% difference), and the first performance parameter will reflect the time that first image505was displayed in the captured video (e.g., a time offset indicating the amount of time that has elapsed from the start of the captured video, a time stamp indicating the actual time the first image was captured). As shown inFIG. 5, the luminance value between first image505and second image510differs greatly due to second image510being approximately halfway through an orientation change. However, the images inFIG. 5are meant to be exemplary of the operations of the method. It should be understood that the second image can typically be an image showing the display in a much earlier stage of rotation due to its occurrence consecutively in time, or near in time, to the first image.

Upon determining when the mobile computing device application started rotating the image displayed on the mobile computing device display, the method can determine when the rotation of the image completed based on analysis of third and fourth images occurring later in time than the first and second images. Test computing device135can determine (420) a first property of a third image of the plurality of images and a first property of a fourth image of the plurality of images. In some examples, the first properties of the third and fourth images are luminance values computed in a substantially similar manner as described above.

Based on a comparison of the luminance values calculated for the third image and the fourth image, test computing device135can determine when the image displayed on mobile computing device105finished rotating. Test computing device135can set (425) a second performance parameter based on a difference between the first property of the third image and the first property of the fourth image. Test computing device135can compare the luminance values computed for the third and fourth images. Upon determining that the computed luminance values are substantially equal, test computing device135can set a second performance parameter indicating a time when the third image was displayed on mobile computing device105. In some embodiments, the first performance parameter is a numeric index indicating which image of the 300 captured images the third image is. In an exemplary embodiment, test computing device135sets the second performance parameter when the luminance values are substantially equal (e.g., differ by 0%). However, other thresholds for the difference between the luminance values of the third and fourth images (e.g., differ by less than 5%, differ by less than 2%, differ by less than 1%) can be used without departing from the spirit of the invention.

FIG. 6is a diagram showing third image605and fourth image610, in accordance with the invention. In the example shown inFIG. 6, third image605is substantially similar or equal to fourth image610. It can be understood that the luminance computed for third image605will be substantially equal in magnitude to the luminance computed for fourth image610. Therefore, the second performance parameter will reflect the time that third image605was displayed in the captured video.

Test computing device135can determine (430) a performance metric based on a difference between the first performance parameter and the second performance parameter. For example, the performance metric can comprise the duration of time it took for an image displayed on the mobile computing device to change from the first orientation to the second orientation. Test computing device135can determine the performance metric based on the difference in time between when the first image was displayed (e.g., T1) and the time the third image was displayed (e.g., T2).

Using the example from above, if the first image is the 20thimage of the 300 captured images, it indicates that the mobile computing device application began rotating the image displayed on the mobile computing device105display at T1=0.0167 seconds per image*20 images=0.334 seconds. If the third image is the 100thimage of the 300 captured images, it indicates that the mobile computing device application completed rotating the image displayed on the mobile computing device105display at T2=0.0167 seconds per image*100 images=1.67 seconds. Accordingly, the performance metric can be the difference between T1and T2, e.g., 1.67 seconds−0.334 seconds=1.336 seconds.

In embodiments in which the first and second performance parameters comprise numerical indexes indicating the image numbers of the first and third images, computations can be performed based on the difference in the image numbers to determine a period of time for the performance metric. In one example in which the images are extracted from the video at a rate of 60 frames per second, the first image (e.g., N1) is the fourth of the 300 captured images (e.g., N1=4), and the third image (e.g., N3) is the 130thimage of the 300 captured images (e.g., N3=130). For this example, the performance metric can be computed based on the difference between the numeric index of the first and third images, N3−N1=130−4=126. This result can be plugged into an equation to convert the difference to a number of seconds: performance delay in seconds=[(126 div 60)+((126 mod 60)/60)]=[(2)+((6)/60)]=2.1 seconds. In some embodiments, test computing device135can add a delay to the performance metric based on a delay between the time arm115rotates mobile computing device105and the time the mobile computing device application begins rotating the image displayed on the screen.

The performance metric can be an indication of the performance of a mobile computing device application. The computed performance metric can be evaluated based on the particular mobile computing device application to determine optimizations that may be needed to the source code of the mobile computing device application, or limitations of the hardware on which the mobile computing device application is executing.

In some examples, the performance of the method can be optimized. For example, in some embodiments, test computing device135can determine the first property of the first image and the first property of the second image based on a subset of the pixels. Test computing device135can compute a pixel luminance for each pixel of the first image as described above. However, instead of computing a luminance value based on the mean of all pixel luminance values, the luminance can be computed based on the luminance of a predetermined number of the pixels of the first image, and using pixels having a predetermined range of pixel luminance values. For example, test computing device135can create a first set of pixels including 25% of the total pixels, where the pixels in the set have the largest computed pixel luminance values of the pixels of the first image. In some embodiments, the first set of can include pixels that have the smallest computed pixel luminance values of the pixels of the first image. In some embodiments, the first set can include between 15% and 30% of the total number of pixels in the first image.

Test computing device can compute a luminance value for the first image based on the mean of the pixel luminance values calculated for the pixels in the first set. Test computing device135can also store position information (e.g., x,y coordinates) about each pixel in the first set.

Referring toFIG. 5, pixels from regions515aand515bcollectively make up first set of pixels515for first image505. Accordingly, the luminance value computed for first image505can be based on the mean of the pixel luminance values calculated for the pixels located within regions515aand515b. Although first set of pixels515comprises regions515aand515b, which are shown as contiguous regions of pixels, it should be understood that first set of pixels515can include pixels from any region of first image505that are within the range of pixels (e.g., top 25% of pixels with largest luminance values), including individual pixels that are not contiguous with any other pixel of first set of pixels515.

Test computing device135can use the position information about the pixels of the first set of pixels to create a second set of pixels including pixels from the second image. For example, each pixel of the second set of pixels can have a position corresponding to a position of a pixel of the first set of pixels. Referring toFIG. 5, pixels from regions520aand520bcollectively make up second set of pixels520for second image510. The position of each pixel in regions520aand520bcorresponds to the position of a pixel in regions515aand515bof first image505.

Test computing device135can compute a pixel luminance for each pixel of the second image as described above. Test computing device can further compute a luminance value for the second image based on the mean of the pixel luminance values calculated for the pixels in the second set of pixels. Test computing device135can then use substantially similar operations as described above to set the first performance parameter according to a difference between the luminance value computed based on the first set of pixels and the luminance value computed based on the second set of pixels.

Similarly, test computing device135can use the position information about the pixels of the first set of pixels to create a third set of pixels including pixels from the third image and a fourth set of pixels including pixels from the fourth image. For example, each pixel of the third set of pixels can have a position corresponding to a position of a pixel of the first set of pixels. Likewise, each pixel of the fourth set of pixels can have a position corresponding to a position of a pixel of the first set of pixels.

Referring toFIG. 6, pixels from regions615aand615bcollectively make up third set of pixels615for third image605. Accordingly, the luminance value computed for third image605is based on the mean of the pixel luminance values calculated for the pixels located within regions615aand615b. Further, pixels from regions620aand620bcollectively make up fourth set of pixels620for fourth image610. Accordingly, the luminance value computed for fourth image610is based on the mean of the pixel luminance values calculated for the pixels located within regions620aand620b. Test computing device135can use substantially similar operations as described above to set the second performance parameter based on a difference between the luminance computed based on the third set of pixels and the luminance computed based on the fourth set of pixels.

For this example, the position of each pixel in regions615aand615b, and regions620aand620brespectively correspond to the position of a pixel in regions515aand515bof first image505inFIG. 5. Accordingly, the method can substantially reduce the amount of storage required for storing the images extracted from the captured video since only a percentage of the pixels from each image are required for each computation. Further, the method can substantially reduce processing times for performing the operations described herein due to the reduced data set on which test computing device135operates.

To provide for interaction with a user, the above described techniques can be implemented on a computing device in communication with a display device, e.g., a CRT (cathode ray tube), plasma, or LCD (liquid crystal display) monitor, a mobile computing device display or screen, a holographic device and/or projector, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse, a trackball, a touchpad, or a motion sensor, by which the user can provide input to the computer (e.g., interact with a user interface element). Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, and/or tactile input.