Patent ID: 12236568

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG.1is diagram of a networked environment100. The networked environment100includes a client-side image processing system102, an image processing system104, and a content provider system106, which communicate over a network108(e.g., the Internet).

The networked environment100verifies that a resource (e.g., document, application interface, webpage, content, etc.) rendered by the client-side image processing system102satisfies a predetermined criterion, for example that the rendered resource includes an attribution (e.g., a visual element) indicative of a content producer (e.g., author, creator, provider, etc.) whose services provided or assisted in providing at least a portion of the resource. The portion of the resource includes content retrieved from the image processing system104. In some implementations, the portion of the resource can include embedded content, such as a widget (e.g., newsfeed), sponsored content, a social networking interface, an in-game purchases interface, or other embedded content. Whilst the below implementation generally describes a criterion associated with including an attribution in a rendered resource, it will be appreciated that the predetermined criterion may be associated with any predetermined display format that may be associated with compliance with requirements associated with the content, for example, requirements to meet legal or regulatory requirements and/or a requirement that content is presented in a particular format.

For example, the image processing system104provides a software platform used by the content provider system106for providing (e.g., generating, assembling, sending, etc.) the resource to the client-side image processing system102. The owner of the image processing system104specifies, for example, that the content provider system106include a visual element that represents the image processing system104. The content provider system106provides the resource (e.g., a webpage, application, program, script or other executable logic, etc.) to the client-side image processing system102. When the resource is rendered by the client-side image processing system102(e.g., caused by the script, program, etc. of the content provider system106), the visual element is displayed in the resource on the user interface (e.g., screen) of the client-side image processing system102. The visual element indicates to the user that at least a portion of the content in the resource is produced by the image processing system104.

The visual element can be rendered in the resource on the client-side image processing system102(e.g., caused by a script, program, or other logic provided by the content provider system106) according to specifications provided by the image processing system104. In some implementations, the image processing system104specifies that the visual element be included in the resource as a condition of providing or assisting in providing at least a portion of the resource for the image processing system104and for the client-side image processing system102. In some implementations, the visual element indicates to a user of the client-side image processing system102that the image processing system104provided or assisted in providing (either directly or indirectly) at least a portion of the resource. In some implementations, the location of the visual element indicates which portion of the resource is being provided by the image processing system104(e.g., a portion including or otherwise proximate the visual element).

The image processing system104verifies that the content provider system106, by generating or otherwise providing the resource to the client-side image processing system102, has included the visual element for being rendered in the resource as agreed upon by the operators of the image processing system104and the content provider system106as a condition of the image processing system104providing (either directly or indirectly) the portion of the resource.

For verification of the rendering of the visual element in the resource, the image processing system104receives data from the client-side image processing system102confirming that the visual element was rendered by the client-side image processing system102. The networked environment100is configured to cause the client-side image processing system102to perform operations for verification as to whether the visual element is rendered in the resource, rather causing the image processing system104(or another system) to perform the verification operations. Locally performing the verification at the client-side image processing system102enables the client-side image processing system102to send verification data122to the image processing system104without including personal data related to a user of the client-side image processing system102, preserving privacy of a user of the client-side image processing system102. In some implementations, the networked environment100causes the client-side image processing system102to send (e.g., transmit) no data related to the client-side image processing system102to the image processing system104that could be used to identify the client-side image processing system102, such as a device identifier, IP address, MAC address, or other identifying data for the purpose of verifying the presence of the visual element in the rendered resource.

Rather, in this example, the networked environment100causes the client-side image processing system102to send only verification data122indicative of whether the visual element is verified as present in the rendered resource. For example, the verification data122includes a bit of data (either a “1” or a “0”) indicating the status of the visual element in the resource. In some implementations, for multiple instances of verification in a single resource, a specified amount of data (e.g., a string of data) can be sent including an identifier for each visual element. The specified amount of data being sent by the client-side image processing system102to the image processing system104reduces bandwidth usage relative to transmitting the data comprising the rendered resource to the image processing system104.

To locally verify the presence of the visual element at the client-side image processing system102(thus preserving privacy and lowering bandwidth usage as describe above), the networked environment100causes the client-side image processing system102to perform one or more image processing operations on data representing the resource when the resource is rendered by the client-side image processing system102. In some implementations, the networked environment100causes the client-side image processing system102to perform the verification at the time of rendering of the resource by the client-side image processing system102. In some implementations, the networked environment100causes the client-side image processing system102to perform the verification at a later time (e.g., after the resource is being displayed to the user). The image processing operations are described in greater detail below in relation toFIGS.3A-3C.

The image processing system104includes one or more processing devices (e.g. a server system) that host content for distribution to one or more other computing systems over the network108. The image processing system104stores one or more software libraries (e.g., software development kits, or “SDKs”) that are accessible by one or content provider systems (e.g., content provider system106) for use in providing content (e.g., text, images, videos, applications, executable scripts, etc.) to one or more client devices (e.g., client-side image processing system102).

For example, the content provider system106can be a website host configured to retrieve newsfeed items for a newsfeed widget on a webpage of the website. The image processing system104provides a software package112including an executable script (e.g., a content embedding script). The software package112can be accessed (e.g., downloaded) by the content provider system106and added to the webpage or otherwise used to build the webpage (e.g., the runtime environment). The software package112is used by the content provider system106in the webpage to cause a client-side image processing system102that is rendering the webpage to send a request for content110from another system (e.g., the image processing system104). The content provider system106retrieves the content116(e.g., the newsfeed items) from the image processing system104or other content producer system. The image processing system104specifies that the webpage should include near the newsfeed a visual element referencing the image processing system104. The client-side image processing system102performs verification operations (as described above and in relation toFIGS.3A-3C) and sends verification data122to the image processing system104.

In another example, the content provider system106can be an application provider (such as a mobile game publisher). The image processing system104owner may desire to include embedded content (e.g., sponsored content, in-app purchasing controls, a social network interface, etc.) in a game being published by the image processing system104. The content provider system106provides application data114(e.g., the game) including the resource to the client-side image processing system102. In this example, when an interface (e.g., the resource) of the game is rendered by the client-side image processing system102that downloaded the application data114provided by the content provider system106, a request for content118is sent by the client-side image processing system102to another device (e.g., image processing system104). The sponsored content (e.g., content120) is retrieved from the other system (e.g., image processing system104) and embedded into the interface of the application (e.g., the mobile game). The sponsored content is displayed by the client-side image processing system102in the interface of the game. The image processing system104specifies that the game interface should include a visual element that represents the image processing system104. In some implementations, the image processing system104specifies that the visual element be located in the resource near (e.g., inside of) the sponsored content. The client-side image processing system102performs verification operations (as described above and in relation toFIGS.3A-3C) and sends verification data122to the image processing system104.

The image processing system104can host content116,120(e.g., text, images, videos, applications, executable scripts, etc.) to be included in resources provided by the content provider system106. In some implementations, the image processing system104sends the content116to the content provider system106, where the content116is assembled with content of the content provider system106before sending to the client-side image processing system102for rendering. In some implementations, the image processing system104sends the content120directly to the client-side image processing system102, such as in response to a request for content118caused by executable logic embedded in an application operating on the client-side image processing system102.

The content provider system106includes one or more computing devices (e.g., a server system) that is operated by a content publisher. While the image processing system104provides content116,120that is typically embedded into a resource, the content provider system106includes systems configured to provide the resources themselves. For example, the content provider system106can include the system of an application publisher, such as for a mobile game, a music application, mapping application, ride service application, or any such application provided by a publisher for running on the client-side image processing system102. For these applications, the content116,120is typically embedded content that is embedded using the software package112(e.g., an SDK) of the image processing system104to enable a functionality with respect to the embedded content in the application. For example, the application publisher may desire to interface the application with a social networking platform provided by the image processing system104. The image processing system104provides the SDK to assist the content provider system106with including the social networking platform access in the application being provided by the content provider system106. In one example, the content provider system106includes a website publisher. The content provider system106may desire to include embedded content in the website, and may desire to embed the content in the website using the software package112(e.g., an SDK) of the image processing system104. In these cases, the verification operations verify that the visual element of the content provider system106is included by the content provider system106to display with the content116,120. The visual element indicates that the image processing system104provided the software package112to the content provider system106for use in generating the resource (e.g., that the platform of the image processing system104is being used).

FIG.2is a block diagram showing components of an image processing system200(e.g., an implementation of networked environment100). The image processing system200includes the image processing system104, the content provider system106, and the client-side image processing system102.

The image processing system104includes a software library206, storage208, a verification module212, and an interface214. The image processing system104includes a platform that can be accessed by one or more content provider systems and from which content can be retrieved and embedded into one or more resources by the client-side image processing system102or the content provider system106. For example, the image processing system104can include an ad exchange, a social network website, a shopping website, a newsfeed provider, or any platform from which content can be retrieved and embedded into a resource (e.g., an application page, a website, etc.). The image processing system104can include a single system that provides both the software library206and other content to be retrieved by the client-side image processing system102or the content provider system106. In some implementations, the image processing system104includes two or more systems, such as a first system that provides the software library206and a second system that provides the content retrieved by a client-side image processing system102. In some implementations, the content produced by the image processing system104is stored at the image processing system104in storage208. The interface214is configured to send and receive data, such as over network108.

The image processing system104includes software library206that enables a content provider system106to retrieve content from the image processing system104and embed the content into a resource provided by the content provider system106. The software library206includes a Software Development Kit that can facilitate development of a resource which retrieves content from the image processing system104(or another system). In some implementations, the software provided by the library includes executable logic, code snippets, etc. that can be combined with the code of a resource to embed content in the resource. In some implementations, the software of the software library206causes the client-side image processing system102to retrieve content (e.g., from the image processing system104) to embed in the resource when the resource is being rendered. In some implementations, the software of the software library206includes embedded content in a package, and does not cause content to be retrieved by the client-side image processing system102.

The image processing system104includes a verification module212that parses data received from other devices (such as the client-side image processing system102) and determines whether the specified visual element was rendered appropriately by the client-side image processing system102. In some implementations, the verification module212receives the verification data indicative of a proper rendering. The verification module212sends a signal to another system (e.g., an ad exchange), which then may take an action based on the verification of the presence of the specified visual element. For example, in an ad context, the ad exchange may only compensate the operator of the content provider system106when the verification data indicates that the visual element has been rendered properly. For example, in a social networking context, a social networking feed can be triggered to resend data based on the verification data indicating that the visual element has not been rendered properly. In some implementations, when the visual element has not been triggered properly, the image processing system104can send a signal to the client-side image processing system102to prohibit the client-side image processing system102from displaying the embedded content.

The client-side image processing system102includes a computing device, such as a laptop, tablet, mobile phone, or other computing device. The client-side image processing system102includes a terminal device on which one or more resources are rendered and displayed. The client-side image processing system102includes an image processing engine210, a data capture engine218, a data transmission device220, storage222, a content embedding engine224, a rendering engine226, and an interface216. The client-side image processing system102can include an application230(e.g., installed on the client-side image processing system102). The application230includes application logic232and an application interface engine234.

The data capture engine218is configured to capture the data that is rendered by the client-side image processing system102(e.g., data to be displayed/being displayed by the client-side image processing system102). In some implementations, the data capture engine218takes a “screenshot” or “snapshot” of the rendered data of a resource. For example, when a resource is rendered, a signal is sent to the data capture engine218to take a screenshot. The data capture engine218captures the data and provides it to the image processing engine210for analysis, as describe below inFIGS.3A-3C. In some implementations, the data capture engine218also receives rendering parameters of the rendered resource. Rendering parameters include the resolution, font sizes, presence/absence of various fields, or other structural information about the resource page to assist the image processing engine210in its analysis of the resource screenshot. For example, the data capture engine218can interrogate the operating system of the client-side image processing system102to determine distortions of the visual element caused by the rendering, such as screen size and stretching or skewing of the resource that can occur when the resource is being rendered for display on the client device. In some implementations, the data capture218can determine whether there has been compression of the visual element, which can introduce further distortions in the visual element. In some implementations, the data capture engine218receives an expected location of the visual element on the screen. For example, the content provider system106can send an expected location of the visual element (e.g., bounding box for each instance of the visual element). The data capture engine218sends the rendering parameters to the image processing engine210in addition to the screenshot of the resource to assist the image processing engine210in determining that the visual element is present. The rendering parameters provided simplify image processing and reduce a latency for verification of the visual element, relative to a latency that occurs for processing the screenshot without rendering parameters. These rendering parameters increase image processing efficiency (relative to image processing efficiency independent of the rendering parameters) by specifying one or more particular portions of the image that need to be processed (e.g., rather than processing the entire image) and by specifying distortions that may be present in the image (which may reduce latency in image processing by pre-identifying those distorted portions, rather than an image processing engine having to identify or test for distortion during processing). In some implementations, where the resource is displayed after verification is confirmed by the image processing engine210, the image processing introduces a latency between rendering the resource and displaying the resource. By reducing the image processing time through the techniques described above, the latency of displaying the resource is reduced while still enabling verification of the presence of the visual element.

The content embedding engine224receives the content (e.g., from the image processing system104or the content provider system106or both). In some implementations, the content is already in storage222as a part of application230. The content embedding engine224configures the resource for rendering by the client-side image processing system102by running the script or other logic provided by the image processing system104. The content embedding engine224assembles the resource content and then sends the resource to the rendering engine226for rendering. The rendering engine226renders the content embedded in the resource, as well as the resource itself.

The application230can include any program for running on the client-side image processing system102that requests content for embedding in the application. For example, application230can include a browser, social network application, mobile game, mapping application, newsfeed, etc. The application230includes application logic232for running the application. The application230includes an application interface engine234that interfaces with the rendering engine226and generates a visualization of the embedded content rendered in the resource.

The image processing engine210receives the visualization from the application interface engine234(e.g., via the data capture engine218). The image processing engine210processes one or more pixels of the generated visualization and verifies that the specified visual element is present in the visualization. In some implementations, the visual element is stored in storage222, such as when the application230is downloaded, and retrieved by the image processing engine210as a pattern to compare the generated visualization (or a portion thereof) against. The image processing engine210processes the generated visualization as describe in relation toFIGS.3A-3Cbelow.

The image processing engine210generates verification data (e.g., verification data122), which can include a single bit or other data that indicates whether the visual element is present in the rendered visualization. The verification data is sent to the data transmission device220which sends the data (via interface216) to the image processing system104(or another remote device) to indicate whether the visual element is present in the resource. In some implementations, the verification is sent as a ping when the resource is rendered. The image processing system104can use anti-spam software to determine whether the ping is genuine or whether verification is being spoofed by the application230. For example, the anti-spam measure can include digital signatures to verify client information, detect emulators, data farming, false signals, etc. These systems are similar to those used to monitor clicks of embedded content (such as ads). In this case, only the verification data is sent to the remote device, and no personal information of the client-side image processing system102or operator of the client device is sent to the remote device. The image processing of the visualization occurs at the client-side image processing system102to protect the privacy of the user of the client-side image processing system102. The personal information included in the screenshot captured by the data capture engine218is thus kept local on the client-side image processing system102and not transmitted over the network108.

However, while no personal data is generally transmitted to the image processing system104, for situations in which the systems discussed here collect and/or use personal information about users, the users may be provided with an opportunity to enable/disable or control programs or features that may collect and/or use personal information (e.g., information about a user's social network, social actions or activities, a user's preferences or a user's current location). In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information associated with the user is removed. For example, a user's identity may be anonymized so that the no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined.

FIGS.3A-3Cshow example screenshots305,310,315of application interface300(e.g., an example of a resource). The image processing engine (e.g., image processing engine210) processes screenshots of interfaces such as interface300(or portions thereof) to determine whether the visual element is present. The image processing engine uses image processing techniques that are available at the client-side image processing system102and that are fast (<100 ms) and reliable. For example, by importing the rendering parameters, the image processing engine uses pre-calculated or pre-determined thresholds for various image processing algorithms. For example, the image processing engine can import preset HSV/HSI thresholds, kernel sizes, filter values, and so forth. By importing the rendering parameters, the image processing engine does not need to calculated compensation metrics for matching the processed image to the retrieved visual element (e.g., visual elements320b-c). The image processing engine can perform one or more image processing algorithms, including feature detection such as by edge detection, Hough transform, scale-invariant feature transforms, segmentation, and so forth.

Turning toFIG.3A, interface300includes content302a,302b,302cfor a game application, such as a mobile game on a smart phone. Content302a-cinclude various modules and controls that enable the user to interact with the interface. The interface300includes embedded content304,306. For example, embedded content304incudes a promotion for a newer version of the game by the content provider system106. For example, embedded content306includes an image including a link to a forum (e.g., a social network) for players of the game. The embedded content304,306can be randomly generated and retrieved by the script that was produced by a content producer system (e.g., image processing system104) and put into the application by content provider system (e.g., content provider system106). The script causes the application to fetch embedded content304,306and embed it into the interface300. Proximate the content304are text visual element308and icon visual element310. Proximate the content306are text visual element308and icon visual element310. The icon310and the text308can be specified by the content producer system to be shown near embedded content provided using the script produced by the content producer system or near embedded content produced by the content producer system. The visual element can include on or more of a logo, text, image, etc. In some implementations, a library of alternate images, logos, text, etc. are provide to the client device and stored for comparison to the visual element. In some implementations, the content provider system106provides the particular visual element being used in the resource. The visual element can be any shape, size, color, orientation, etc.

Turning toFIG.3B, a screenshot of a portion of interface300is processed by an image processing engine312(e.g., image processing engine210). The methods for processing the screenshot include methods that are simple such that processing is less than or approximately equal to 100 ms. In some implementations, the processing time can be up to 250 ms. The image processing does not require knowledge of the structure of the resource to determine the presence or absence of the visual element. The image processing is less invasive than parsing the structure of the resource to determine whether data representing the visual element are present in the resource. The image processing is faster than parsing the structure of the resource to determine whether data representing the visual element are present in the resource. The image processing can be simplified using information received from the content provider system that provided the resource, such as where on the interface300the visual element should appear (e.g., a bounding box for the visual element). Compression and resolution data can be retrieved as described above to further reduce image processing time and increase the accuracy of the image processing. For example, compression and rendering parameters, operating system rendering distortions, etc. can be used to tune image processing thresholds, reduce complexity of the image processing algorithm, increase accuracy of the verification, and reduce processing time. In some implementations, these techniques are used because machine learning is not available at the client device for verifying the visual element, as no training data are available.

To process the logo310, the image processing engine312receives dimensions of bounding box314that indicate a region of the resource on which the image processing should occur. The region of the interface inside bounding box314is processed using one or more image processing algorithms, such as convolution with a static kernel. The rendering parameters318(e.g., retrieved from the operating system by the data capture engine218) are used to assist the image processing engine312. The image processing engine312retrieves the specified visual element320bis retrieved from storage320aand compares the specified visual element320bto the processed image316. The image processing engine312generates verification data322and sends the data to a remote device, such as the content producer324.

Turning toFIG.3C, interface300is shown which includes text308as a visual element. Similar to the image processing of the logo310ofFIG.3B, to process the text308, the image processing engine312receives dimensions of bounding box314that indicate a region of the resource on which the image processing should occur. The region of the interface inside bounding box314is processed using one or more image processing algorithms, such as convolution with a static kernel. The rendering parameters318(e.g., retrieved from the operating system by the data capture engine218) are used to assist the image processing engine312. The image processing engine312retrieves the specified visual element320cis retrieved from storage320aand compares the specified visual element320bto the processed image316. The image processing engine312generates verification data322and sends the data to a remote device, such as the content producer324. In some implementations, the image processing engine312receives data indicating that the visual element316comprises text, and performs optical character recognition (OCR) image processing on the visual element316.

In some implementations, in addition to or alternative to image processing to verify that data is rendered, “audio processing” can be performed to verify that that a predetermined audio output by a device is actually being output by the device in accordance with the predetermination. For example, in a voice activated device (e.g., the client device in this case), content providers might provide plugins or modules to enable a user of the client device communicate directly with services of the content provider. The image processing system can enable the content providers to integrate these services with interface of the voice activated device so that the user can access the content provider's content via the voice activated device. The image processing system can monitor generated audio data by the voice activated device to verify that content provided by the image processing system (or any other device) are correctly attributed in the audio data (e.g., speech data) generated by the voice activated device. In some implementations, the device need not be a voice-activated device, but instead can be any device provided an auditory output along with (e.g., in response to a request for) data from the content provider. The image processing system can verify that a particular word is spoken, a particular tone (e.g., theme, etc.) is played, and so forth in along with data provided by the content provider. In some implementations, the process for audio processing for verification of rendered data is similar to the image processing examples describe through this document, where image processing techniques are replaced with signal processing techniques for audio output. For example, techniques including heterogeneous parameter sets that mix absolute spectral information with dynamic, or time-derivative, spectral information, Hidden Markov models, and so forth can be used.

FIG.4is flow diagram of actions400taken by systems of the image processing system (e.g. networked environment100). The content provider system106requests (402) software (such as an SDK) from the image processing system104. The image processing system104determines (404) a specified visual element (or library of elements) and sends (406) the requested software to the image processing system104along with the specified element(s). The specified element is to be shown with any content embedded using the software provided by the image processing system104in resources provided to the client-side image processing system102by the image processing system104. The content provider system106builds (408) the resource with the software, including the specified visual element. The image processing system104sends (410) the resource to a client-side image processing system102. The resource can be an application or part of an application, a webpage or other document, and so forth. The client-side image processing system102receives (412) the resource. When the resource is activated (e.g., at runtime), the client-side image processing system102requests (414) content for embedding into the resource, such as from the image processing system104. The image processing system104determines (416) which content to provide (e.g., via an auction or other content evaluation mechanism). The image processing system104sends (418) the content to the client-side image processing system102. The client-side image processing system102embeds (420) the content in the resource. The client-side image processing system102renders (422) the resource. The client-side image processing system102determines (424) the presence of the specified visual element, such as using the image processing engine. The client-side image processing system102displays (426) the resource and sends (428) the verification data to be received (430) by the image processing system104. In some implementations, the verification data is sent (428) before the resource is displayed (426) by the client-side image processing system102.

FIG.5is a flow diagram showing actions500of the image processing system. The image processing system embeds (502) the content including the visual element in a resource, such as a resource provided by the content provider system106. The image processing system renders (504) the resource and the embedded content. The image processing system determines (506) rendering parameters. The image processing system retrieves (508) the specified visual element. The image processing system processes (510) the generated visualization. The image processing system determines (512) the presence of the specified visual element. The image processing system transmits (514) verification data indicative of the determination results.

FIG.6is a block diagram of computing devices600,650that may be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. Computing device600is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device650is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. Additionally, computing device600or650can include Universal Serial Bus (USB) flash drives. The USB flash drives may store operating systems and other applications. The USB flash drives can include input/output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

Computing device600includes a processor602, memory604, a storage device606, a high-speed interface608connecting to memory604and high-speed expansion ports610, and a low speed interface612connecting to low speed bus614and storage device606. Each of the components602,604,606,608,610, and612, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor602can process instructions for execution within the computing device600, including instructions stored in the memory604or on the storage device606to display graphical information for a GUI on an external input/output device, such as display616coupled to high speed interface608. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices600may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory604stores information within the computing device600. In one implementation, the memory604is a volatile memory unit or units. In another implementation, the memory604is a non-volatile memory unit or units. The memory604may also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device606is capable of providing mass storage for the computing device600. In one implementation, the storage device606may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory604, the storage device606, or memory on processor602.

The high speed controller608manages bandwidth-intensive operations for the computing device600, while the low speed controller612manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In one implementation, the high-speed controller608is coupled to memory604, display616(e.g., through a graphics processor or accelerator), and to high-speed expansion ports610, which may accept various expansion cards (not shown). In the implementation, low-speed controller612is coupled to storage device606and low-speed expansion port614. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device600may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server620, or multiple times in a group of such servers. It may also be implemented as part of a rack server system624. In addition, it may be implemented in a personal computer such as a laptop computer622. Alternatively, components from computing device600may be combined with other components in a mobile device (not shown), such as device650. Each of such devices may contain one or more of computing device600,650, and an entire system may be made up of multiple computing devices600,650communicating with each other.

Computing device650includes a processor652, memory664, an input/output device such as a display654, a communication interface666, and a transceiver668, among other components. The device650may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components650,652,664,654,666, and668, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

The processor652can execute instructions within the computing device650, including instructions stored in the memory664. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. Additionally, the processor may be implemented using any of a number of architectures. For example, the processor410may be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor. The processor may provide, for example, for coordination of the other components of the device650, such as control of user interfaces, applications run by device650, and wireless communication by device650.

Processor652may communicate with a user through control interface658and display interface656coupled to a display654. The display654may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface656may comprise appropriate circuitry for driving the display654to present graphical and other information to a user. The control interface658may receive commands from a user and convert them for submission to the processor652. In addition, an external interface662may be provided in communication with processor652, so as to enable near area communication of device650with other devices. External interface662may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

The memory664stores information within the computing device650. The memory664can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory674may also be provided and connected to device650through expansion interface672, which may include, for example, a SIMM (Single In-Line Memory Module) card interface. Such expansion memory674may provide extra storage space for device650, or may also store applications or other information for device650. Specifically, expansion memory674may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory674may be provided as a security module for device650, and may be programmed with instructions that permit secure use of device650. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory664, expansion memory674, or memory on processor652that may be received, for example, over transceiver668or external interface662.

Device650may communicate wirelessly through communication interface666, which may include digital signal processing circuitry where necessary. Communication interface666may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver668. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module670may provide additional navigation- and location-related wireless data to device650, which may be used as appropriate by applications running on device650.

Device650may also communicate audibly using audio codec660, which may receive spoken information from a user and convert it to usable digital information. Audio codec660may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device650. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device650.

The computing device650may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone680. It may also be implemented as part of a smartphone682, personal digital assistant, or other similar mobile device.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Although a few implementations have been described in detail above, other modifications are possible. Moreover, other mechanisms for the image processing system may be used. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.