Patent Description:
This specification describes technologies relating to data processing, and generating non-falsifiable attestation tokens to verify human interaction with content while protecting user privacy.

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving a rendering notification of a rendered element defined in an active window on a client device, receiving a declaration of the rendered element defined in the active window on the client device, detecting interaction with the rendered element at the client device, determining, based on the received declaration, whether the interaction occurred at a declared location of the rendered element within the active window, and processing the interaction based on the determination of whether the interaction occurred at the declared location. Processing the interaction includes: in response to determining that the interaction occurred at the declared location of the rendered element: capturing a screenshot of the active window on the client device, verifying a visual appearance of the rendered element in the screenshot with a declared appearance of the rendered element, and generating an interaction attestation for the interaction with the rendered element, thereby validating the interaction. In response to determining that the interaction did not occur at the declared location of the rendered element, refraining from generating the interaction attestation for the interaction with the rendered element.

Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

These and other embodiments can each optionally include one or more of the following features. In some implementations, the rendering notification includes one or thereby validating the interaction. In response to determining that the interaction did not occur at the declared location of the rendered element, refraining from generating the interaction attestation for the interaction with the rendered element.

These and other embodiments can each optionally include one or more of the following features. In some implementations, the rendering notification includes one or more of i) a location of the rendered element in the active window, ii) a stable ID and URI of the rendered element and iii) a site or domain that rendered the rendered element.

In some implementations, receiving the declaration of the rendered element includes receiving a verified appearance of the rendered element. The verified appearance can include a unique stable ID and/or stable URL and a verified shape.

In some implementations, verifying a visual appearance of the rendered element in the screenshot with a declared appearance of the rendered element includes utilizing an image matching algorithm.

In some implementations, detecting the interaction includes receiving a selection signal specifying an XY coordinate of a user click within the active window.

In some implementations, the methods further includes aggregating the interaction attestations for multiple interactions with multiple instantiations of the rendered element. The interaction attestation can include one or more of i) a stable ID and/or URI of the declared appearance, ii) a timestamp of the interaction, iii) a device integrity token, iv) a digital signature by the client device, v) XY coordinates of the interaction, and vi) a client device public key. The stable ID and/or URI of the declared appearance can include a user intent of the interaction, where the user intent can correspond to an action conveyed by the appearance of the rendered element.

In some implementations, the methods further include: in response to the interaction attestation including a user intent, modifying a report of the interaction based on the user intent. The methods can further include: in response to determining that the interaction is different to the user intent, determining that the rendered element is a false rendered element and triggering a false interaction response. The methods can further include, in response to triggering the false interaction response, intercepting the pathway of the client device to a landing page of the false rendered element.

In some implementations, in response to determining that the interaction did not occur at the declared location of the rendered element, the methods include triggering an false interaction response.

Verifying that a real user has interacted with a rendered element, e.g., a button, a thumbs up/down, etc., at a remote client device can preserve the authenticity of a ranking mechanism for a digital component, e.g., a news article, and prevent bad actors from manipulating a ranked set of digital components. By validating the appearance of a rendered component at a remote client device using third-party validation, the system can ensure that an actual user is interacting with the intended rendered element, e.g., button, rather than a hidden or manipulated element that may be trying to launch a secondary action or hijack the interaction altogether. This subject matter can be utilized to rapidly identify fake rendered elements distributed through an online system using validated attestation tokens that preserve user privacy. Using the validated attestation tokens, the system can ensure that a detected interaction was performed at a remote client device by an actual user, and that the user was actually provided the content at the time of the detected interaction. This type of verification uses verifiable digital signatures created by one or more devices and which cannot be falsified by a human operator to validate interactions with content at remote client devices.

By ensuring that the rendered user interface (UI) elements being processed are valid prior to the processing of secondary actions, e.g., linking to a landing page, adjusting underlying ranking mechanisms associated with the rendered element, etc., the processing, memory, and resources being used to provide content to end-users is reduced because resources are not being utilized to store, distribute, or present false information associated with the fake rendered elements. Additionally, using attestation tokens can assist in classifying rendered elements presented with digital components as false in a more reliable and efficient manner, reducing the resources required to store and distribute misleading information. For example, the attestation tokens are configured to ensure that the interactions with the rendered elements are made by users that were actually exposed to and/or clicked on actually rendered elements, while still preserving the user's privacy. As such, the interactions can be confirmed to be legitimate rather than being made by malicious actors (e.g., entities trying to harm a particular content distributor). As such, resources are not wasted processing illegitimate false rendered element alerts, and the resulting metrics and/or reports providing information about digital components that contain false rendered elements are made more accurate and are more efficiently created because of the ability to validate each rendered element as legitimate or illegitimate using the attestation tokens. Furthermore, the techniques described throughout this document enable systems to warn users when they are being presented with content that may be false or misleading, thereby providing the users with the ability to avoid misleading or malicious content.

In general, this document relates to computer implemented methods and systems to validate interactions with false rendered elements, e.g., misleading and/or invisible buttons, presented with digital components, e.g., video clips, audio clips, multimedia clips, images, text, or another unit of content, using verified rendered element appearance and interaction attestation. A user's interaction with a rendered element in an active window on a client device, e.g., in an application environment or website, can be reported by the operating system or other trusted application or web browser, along with information related to the appearance of the rendered element in the active window, e.g., a screenshot of the active window. The appearance of the rendered elements can be verified by a third party and include a declaration of appearance accessible by the application and/or web browser presenting the rendered element. The rendered element can be validated using the third party declaration of the appearance of the rendered element and a screenshot of the active window of the client device, to verify that the interaction by the user was intentional. Validating the interaction as a human interaction (rather than an automated, or bot, interaction) and the rendered element as not a false rendered element can limit, modify, or halt false rendered elements from being used to distribute digital components that present false information, as discussed throughout this document.

Operating systems (OS), e.g., for applications, and browsers can validate user interactions with rendered elements in a privacy-preserving manner, such that the false rendered elements can be identified using a central aggregation server(s) that implements privacy-preserving aggregation measurements. Aggregated false rendered element reports can be generated from the validated user interactions with the rendered elements, which can be utilized to alert a user to potential false rendered elements present in an active window on the client device. Systems and methods for validating user interaction with rendered elements are described in further detail below with reference to <FIG>, <FIG>, and <FIG>.

<FIG> is a block diagram of an environment <NUM> in which a digital component system <NUM> distributes digital components. The example environment <NUM> includes a data communication network <NUM>, such as a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof. The network <NUM> connects client devices <NUM>, publishers <NUM>, websites <NUM>, the digital component distribution system <NUM>, a detection system <NUM>, an aggregation system <NUM>, and a reporting system <NUM>. The example environment <NUM> may include many different client devices <NUM>, publishers <NUM>, and websites <NUM>. In some implementations, the environment <NUM> can also include multiple digital component distribution systems <NUM>.

A website <NUM> is one or more resources <NUM> associated with a domain name and hosted by one or more servers. An example website is a collection of web pages formatted in HTML that can contain text, images, multimedia content, and programming elements, such as scripts. Each website <NUM> is maintained by a publisher <NUM>, which is an entity that controls, manages and/or owns the website <NUM>.

A resource <NUM> is any data that can be provided over the network <NUM>. A resource <NUM> is identified by a resource address, e.g., a Universal Resource Locator (URL), that is associated with the resource <NUM>. Resources include HTML pages, word processing documents, and portable document format (PDF) documents, images, video, and feed sources, to name only a few. The resources can include content, such as words, phrases, images and sounds, that may include embedded information (such as meta-information in hyperlinks) and/or embedded instructions (such as scripts).

A client device <NUM> is an electronic device that is capable of communicating over the network <NUM>. Example client devices <NUM> include personal computers, mobile communication devices, e.g., smart phones, and other devices that can send and receive data over the network <NUM>.

A client device <NUM> typically includes applications <NUM>, such as web browsers and/or native applications, to facilitate the sending and receiving of data over the network <NUM>. A native application is an application developed for a particular platform or a particular device. Publishers <NUM> can develop and provide, e.g., make available for download, native applications to the client devices <NUM>. In some implementations, the client device <NUM> is a digital media device, e.g., a streaming device that plugs into a television or other display to stream videos to the television. The digital media device can also include a web browser and/or other applications that stream video and/or present resources.

A web browser can request a resource <NUM> from a web server that hosts a website <NUM> of a publisher <NUM>, e.g., in response to the user of the client device <NUM> entering the resource address for the resource <NUM> in an address bar of the web browser or selecting a link that references the resource address. Similarly, a native application can request application content from a remote server of a publisher <NUM>.

Some resources <NUM>, application pages, or other application content can include digital component slots for presenting digital components with the resources <NUM> or application pages. As used throughout this document, the phrase "digital component" refers to a discrete unit of digital components or digital information (e.g., a video clip, audio clip, multimedia clip, image, text, or another unit of content). A digital component can electronically be stored in a physical memory device as a single file or in a collection of files, and digital components can take the form of video files, audio files, multimedia files, image files, or text files and include advertising information, such that an advertisement is a type of digital component. For example, the digital component may be content that is intended to supplement content of a web page or other resource presented by the application <NUM>. More specifically, the digital component may include digital content that is relevant to the resource content (e.g., the digital component may relate to the same topic as the web page content, or to a related topic). The provision of digital components by the digital component distribution system <NUM> can thus supplement, and generally enhance, the web page or application content.

When the application <NUM> loads a resource <NUM> (or application content) that includes one or more digital component slots, the application <NUM> can request a digital component for each slot from the digital component distribution system <NUM>. The digital component distribution system <NUM> can, in turn request digital components from digital component providers <NUM>. The digital component providers <NUM> are entities that provide digital components for presentation with resources <NUM>.

In some cases, the digital component distribution system <NUM> can also request digital components from one or more digital component partners <NUM>. A digital component partner <NUM> is an entity that selects digital components <NUM> (not shown) on behalf of digital component providers <NUM> in response to digital component requests.

The digital component distribution system <NUM> can select a digital component for each digital component slot based on various criteria. For example, the digital component distribution system <NUM> can select, from the digital components received from the digital component providers <NUM> and/or the digital component partners <NUM>, a digital component based on relatedness to the resource <NUM> (or application content), performance of the digital component (e.g., a rate at which users interact with the digital component), etc. The digital component distribution system <NUM> can then provide the selected digital component(s) to the client device <NUM> for presentation with the resource <NUM> or other application content.

When the application <NUM> presents a digital component, the application <NUM> (which could be a browser) can store impression data <NUM> for the presentation of the digital component. The impression data <NUM> for a particular presentation of a digital component can include a URL or domain of a landing page for the digital component (e.g., a page linked to by the digital component and when the user clicks or interacts with the digital component, the application/browser presents the page to the user). In some implementations, the impression data <NUM> for the particular presentation of the digital component can include a phone number, e.g., for click-to-call advertisements, or store addresses, e.g., for in-store visit advertisements.

The impression data <NUM> for the particular presentation of the digital component can further include one or more identifiers for the digital component, event-level data associated with the impression, an expiration time that specifies when the impression data is to be deleted from the client device <NUM>, and/or a reporting URL or domain to which conversion reports for the digital component are to be sent. This data can be provided by the digital component, e.g., as metadata of the digital component or an anchor tag of the digital component. As described in more detail below, rather than storing, or in addition to storing, raw impression data for each impression of a digital component, the application <NUM> can store impression data that is blindly signed by the detection system <NUM>.

The application <NUM> can also store conversion data <NUM> in response to a conversion being detected. A conversion for a digital component is the completion of a specified user action after the digital component is presented to and/or interacted with (e.g., clicked or tapped) by the user. The conversion data <NUM> for a conversion can include data that indicates a type of the conversion as some conversions can have multiple types. The type of a conversion can specify a subsequent action after the conversion is completed. For example, a conversion may be the addition of an item to an online shopping cart. In this example, a first type of conversion may be the addition of the item without checking out (e.g., not completing a purchase) and a second type of conversion may be checking out. Other examples of conversion include, phone call to the advertiser, installation of an application, a visit to a local store owned by the advertiser, or the like. As described in more detail below, the conversion data for a conversion can be a set of one or more bits that indicate the type of conversion and the application can store conversion data that is blindly signed by the digital component distribution system <NUM> (or another appropriate system).

In some implementations, a conversion for a digital component includes interaction data for a user interaction with a rendered element presented with the digital component. The interaction data <NUM> including a user interaction with the rendered element can include data that indicates a type of interaction, e.g., a click, swipe, finger-up/finger-down, etc., with the rendered element, e.g., a button. Interaction with rendered elements is described below in further detail with reference to <FIG> and <FIG>.

The detection system <NUM> can evaluate detection signals received from the client device <NUM> to determine whether an impression, interaction, and/or conversion is valid or invalid. The detection signals can be dependent on the application and can vary in different implementations. The application <NUM> (or the operating system on client device <NUM>) can include an API that enables a digital component to make a call to the application <NUM> (or the operating system on client device <NUM>) to collect the detection signals and provide the detection signals to the detection system <NUM>.

In addition to the detection signals, the application <NUM> can send blinded impression data for the impression to the detection system <NUM>. If the detection system <NUM> determines that the impression is valid (e.g., that identified content was actually presented by a client device), the detection system <NUM> can sign the blinded impression data and provide the signed blinded impression data to the application <NUM>. Example techniques for generating blinded impression data and signing the blinded impression data are described below.

The aggregation system <NUM> can generate aggregated network measurements based on data received from client devices <NUM>. In the following description, the techniques for generating and providing measurement data elements is described as being performed by the application <NUM>, which can be a web browser or native application. However, in some implementations, the operating system of the client device <NUM> can generate and send the measurement data elements. In such implementations, the web browser(s) and application(s) on the client device <NUM> can be configured to report impressions and conversions to the operating system. The operating system can perform each of the operations for reporting the impressions and conversions described below as being performed by the application <NUM>.

The application <NUM> on the client devices <NUM> can provide, to the aggregation system <NUM>, measurement data elements that include encrypted data that represents network data. The network data can include impression data and/or conversion data for each conversion. For example, the application <NUM> can generate and send to the aggregation system <NUM>, a measurement data element for each impression, user interaction, and/or conversion that takes place at the client device <NUM>. The aggregated network measurements can include, for each of one or more digital components, a total number of impressions, user interactions, and/or conversions for the digital component and, optionally, a rendered element presented with the digital component, across multiple client devices <NUM>.

The application <NUM> can use an (t, n) threshold scheme to generate the data in a measurement data element. In some implementations, when the application <NUM> detects a conversion or receives conversion data for a conversion, the application <NUM> generates a group key (e.g., a polynomial function) based on impression data and conversion data for the conversion. The application can then generate a group member key that represents a portion of the group key and that can be used to regenerate the group key only when a sufficient number of group member keys for the same impression and conversion pair are received. In this example, the measurement data element for a conversion can include the group member key generated by the application and a tag that corresponds to the impression and conversion pair. Each unique impression and conversion pair can have a corresponding unique tag so that the aggregation system <NUM> can aggregate the measurement data elements for each impression and conversion pair using its tag.

In some implementations, when the application <NUM> detects an interaction and/or conversion or receives interaction and/or conversion data for an interaction and/or conversion, the application <NUM> generates a group key (e.g., a polynomial function) based on impression data, interaction data, and conversion data for the interaction and/or conversion. The application can then generate a group member key that represents a portion of the group key and that can be used to regenerate the group key only when a sufficient number of group member keys for the same impression and interaction/conversion pair are received. In this example, the measurement data element for an interaction/conversion can include the group member key generated by the application and a tag that corresponds to the impression and interaction/conversion pair. Each unique impression and interaction/conversion pair can have a corresponding unique tag so that the aggregation system <NUM> can aggregate the measurement data elements for each impression and interaction/conversion pair using its tag.

In a (t, n)-threshold encryption scheme, the aggregation server <NUM> would need to receive at least t group member keys for the same impression and interaction/conversion pair to be able to decrypt the impression and conversion and/or interaction data. If less than t group member keys are received, the aggregation server <NUM> cannot decrypt the impression and conversion and/or interaction data. Once at least t measurement data elements for the same impression and conversion and/or interaction pair are received from client devices <NUM>, the aggregation system <NUM> can determine the group key from the at least t group member keys and obtain the impression and conversion and/or interaction data from the group key.

The aggregation system <NUM> can determine the quantity of the conversions for an impression and conversion pair based on the number of measurement data elements received that includes impression data, interaction data, and/or conversion data for the impression and interaction/conversion pair. For example, after obtaining the impression, interaction, and conversion data using the at least t group member keys, the aggregation system <NUM> can determine, as the quantity of conversions, a count of the number of the group member keys received for the impression and interaction/conversion pair. The aggregation system <NUM> can report the impression data, the interaction data, the conversion data, and the quantity of the interactions and/or conversions to the reporting system <NUM>, which may be implemented as a reporting domain that corresponds to a reporting URL of the digital component that, in turn, corresponds to the impression and conversion data.

Digital component system <NUM> can distribute digital components from various digital component providers <NUM> to client devices <NUM>. Digital components can include rendered elements e.g., buttons, where users can interact with the rendered elements. Rendered elements can be validated using declarations of appearance and screenshots captured of the appearance of the rendered element in the active window of the client device <NUM>. Furthermore, interactions with the rendered elements can be verified and aggregated in order to protect the privacy of the reporting end-user as well as preserving the interests of the digital component providers <NUM>.

<FIG> and <FIG> are block diagrams of environments in which rendered element interactions are validated. A false rendered element refers to a rendered element, e.g., a button or otherwise selectable feature, presented with a digital component that is invisible or misleading to a user. In some implementations, a false rendered element can trigger an interaction that is not intended by the user. In one example, a hidden button (e.g., a code-defined interaction zone for the button that does not include a graphical visualization of the button presented on screen) over a "close window" button can launch a request for another application or webpage instead of closing the current window or application, as intended by the user by the user's selection of the "close window" button. In another example, a misleading rendered element can appear as a "thumbs up" icon but a user selection of the misleading rendered element can trigger a "down vote" function rather than the intended "up vote" function.

<FIG> is a block diagram of an environment <NUM> in which rendered element interactions are validated. Interaction validation system <NUM> is configured to receive a rendering notification <NUM> and an interaction alert <NUM> from an operating system <NUM> on a client device <NUM>. Interaction validation system <NUM> is further configured to receive a declaration <NUM> of a verified appearance of a rendered element <NUM> in an active window <NUM> of an application <NUM>. Additionally, interaction validation system <NUM> is configured to receive, from the operating system <NUM> or another trusted application <NUM> of the client device <NUM>, a screenshot <NUM> of the active window <NUM> displayed on the client device <NUM>.

In some implementations, one or more of the processes described herein as performed by the interaction validation system <NUM> can be performed by the operating system <NUM> or another trusted application <NUM> on the client device <NUM>.

Interaction validation system <NUM> can include an interaction determination module <NUM>, a visual analysis module <NUM>, and an attestation generator <NUM>. Each of the interaction determination module <NUM>, visual analysis module <NUM>, and attestation generator <NUM> can include computer storage readable media encoded with the functionality to perform the tasks described herein with reference to the respective module and/or one or more processors configured to perform the tasks described. Interaction determination module <NUM> is configured to receive a rendering notification <NUM> and an interaction alert <NUM> as input. Rendering notification <NUM> can include a location of the rendered element <NUM> in the active window <NUM>, a stable identification (stable ID) and/or uniform resource identifier (URI) of the rendered element <NUM>, and a site or domain that rendered the rendered element <NUM>.

A rendered element <NUM> is an element that is visible in an active window <NUM>, e.g., an active window on a client device <NUM>. A rendering notification <NUM> can be generated by an application notifying the operating system <NUM> that a rendered element <NUM> is visible in the active window <NUM>. In another embodiment, rendering notification <NUM> can be generated by a webpage/JavaScript notifying the web browser that the rendered element <NUM> is visible in the active window of a web page, e.g., as described with reference to <FIG> below.

Rendered elements <NUM> can be requested by the operating system <NUM> as a part of, or in addition to, a request for digital component <NUM>. The request for digital components can include a rendered element <NUM>, e.g., a news post with share, vote up, and vote down buttons. The digital component provider <NUM> can provide a schema for generating the rendered element <NUM> and rendering control including a digital signature for the rendered element <NUM>.

A rendering notification <NUM> can be generated each time the appearance of the active window changes, e.g., by the user scrolling, resizing, or otherwise changing a current view of the active window. Tracking a location of the rendered element <NUM> within the active window <NUM> can include tracking user scrolling, zoom functionality, etc., where user interaction with scrolling functionality, zoom functionality, etc., will trigger a signal to update the rendering notification <NUM>. In some implementations, the rendering notification can be generated by a script, such as a mutation observer, or another script that can detect changes to a user interface (e.g., a DOM of a web page or a structure of a native application user interface).

Referring back to <FIG>, an interaction <NUM> with a rendered element <NUM> can include but is not limited to a user selection, e.g., a click, swipe, pinch, mouse hover, or other motion indicating a selection of the rendered element within an area declared to be occupied by the rendered element <NUM>. Interaction <NUM> can be a user scrolling through at least a portion of a digital component <NUM>, e.g., scrolling to a bottom of a news article. Interaction <NUM> can be a user passively allowing playback of a digital component <NUM>, e.g., playback of an audio, video, or multi-media file. In one example, an interaction <NUM> is detected when a user's finger touches down on and/or lifts up from a touchscreen of a client device <NUM>.

Additionally, the interaction determination module <NUM> receives declaration <NUM>. Receiving the declaration <NUM> of the rendered element <NUM> can include receiving a verified appearance of the rendered element <NUM>. A verified appearance of the rendered element <NUM> can include a unique stable ID, e.g., a cryptohash of a resource file, or a cryptohash of rendered RGB pixel values, and/or stable URI, and a verified shape of the rendered element <NUM>. In one example, a verified shape includes dimensions and a type of shape, e.g., square, rectangle, circle, etc. In another example, a verified shape includes a range of dimensions and a general category and/or descriptive identifiers of the shape, e.g., a thumbs up shape, a stop sign shape, a checked box shape. In another example, the verified shape may be represented by a bitmap (2D array), where <NUM> represents "part of shape" and <NUM> otherwise. Such a bitmap can represent an arbitrary complex shape.

Declaration <NUM> is not a user specific, e.g., not specific to a particular user or particular client device <NUM>. The declared appearance of the rendered element <NUM> is consistent for a sufficiently large group of users on various client devices. For example, the declared appearance includes ranges of dimensions and color palette to accommodate variations that are possible between different client devices, display characteristics, and user preferences. In another example, the declared appearance includes multiple languages and other region variations possible for the rendered element <NUM>.

Declaration <NUM> including a declared appearance of the rendered element is independently verified by a third-party as not misleading. In one example, rendered elements in an application can be verified by an application store or other provider that lists the application, for example, as a part of a review and approval process for providing the application. In another example, rendered elements in a web resource can be verified by a web browser vendor and/or privacy group for website publishers.

Third-party reviewers can determine that the rendered buttons satisfy a review criteria, e.g., that an interaction with a rendered button results in an action matching a user intent, and digitally sign the rendered element resource, e.g., a png or bitmap file, that can be used to verify the rendered element. The user intent may therefore correspond to an action conveyed by the appearance of the rendered element. In some implementations, third-party reviewers can generate an enabled list of approved appearances that a web browser can access.

Interaction determination module <NUM>, determines, based on the received declaration <NUM>, whether the interaction <NUM> of the interaction alert <NUM> occurred at a declared location of the rendered element <NUM> within the active window <NUM> of the client device <NUM>. Detecting the interaction <NUM> has occurred at a declared location of the rendered element <NUM> can include receiving, from the operating system <NUM> or other trusted application <NUM>, a selection signal specifying an XY coordinate of a user click or other form of user interaction with the rendered element <NUM> within the active window <NUM>. In some implementations, detecting the interaction <NUM> at the declared location of the rendered element includes receiving a range of X'-X" and a range of Y'-Y" coordinates for the declared area occupied by rendered element in the active window <NUM>, comparing the X coordinate of the interaction <NUM> and the Y coordinate of the interaction <NUM> to the ranges of X'-X" and Y'-Y", and determining whether the X coordinate of the interaction <NUM> and the Y coordinate of the interaction <NUM> are within the ranges of X'-X" and Y'-Y", respectively.

Based on the determination that the interaction <NUM> occurred at a declared location of the rendered element <NUM> within the active window <NUM>, the interaction determination module <NUM> provides a confirmation of interaction as output to the visual analysis module <NUM>. In one example, for an application <NUM>, the operating system <NUM> will determine that the interaction <NUM> occurred at the declared location of the rendered element by examining whether the interaction <NUM> occurred within an area declared to be occupied by the rendered element <NUM>. In another example, as depicted in <FIG>, for a web resource <NUM>, the browser <NUM> will examine whether the interaction <NUM> occurred within an area declared to be occupied by the rendered element <NUM>.

The visual analysis module <NUM> is configured to receive as input, from the operating system <NUM>, a screenshot <NUM> of the active window of the client device <NUM> including the rendered element <NUM>. Additionally, the visual analysis module <NUM> is configured to receive as input a declaration of the rendered element <NUM> from a repository of declarations <NUM> for rendered elements. The visual analysis module <NUM> verifies a visual appearance of the rendered element <NUM> in the screenshot <NUM> with a declared appearance of the rendered element <NUM> from declaration <NUM> of the rendered element <NUM>.

In some implementations, verifying the visual appearance of the rendered element includes utilizing one or more image matching algorithms to compare the declared appearance to the screenshot <NUM>. For example, the image matching algorithms can extract image features (e.g., pixel data) of the rendered element, identify a stored set of image features corresponding to the rendered element (e.g., as previously declared by a publisher of the rendered element), and compare the extracted image features to the stored image features to determine whether there is a match between the extracted image features and the stored image features. When verifying the visual appearance of the rendered element, variations due to rendering difference, e.g., due to scaling, screen hue/brightness/contrast adjustment, font used to render the element text, etc., can be accounted for. For example, when comparing the screenshot <NUM> against the declaration <NUM> for the rendered element <NUM>, an image matching algorithm can allow for small variations in rendering with algorithms to normalize size, contrast, brightness, etc., by using, for example, variants of a two-dimensional normalized cross correlation algorithm or another appropriate algorithm.

In some implementations, a declaration <NUM> can include "true" rendering values that are not subject to hue masks, aspect skewing, or other similar rendering changes that can occur when an element is rendered on a particular client device or web resource. In this embodiment, the operating system and/or browser can utilize the declaration <NUM> without necessarily accounting for any small variations in rendering. In one example, an operating system can have rendering architecture including a device-independent bitmap.

In some implementations, verification of the visual appearance of the rendered element <NUM> and the declared appearance of the rendered element includes meeting a threshold of similarity, e.g., rendered appearance of the rendered element is within a range of variability from the declared appearance of the rendered element.

Based on the verification that the visual appearance of the rendered element <NUM> is matching and/or within a threshold of similarity, the visual analysis module <NUM> provides a verification of interaction as output to the attestation generator <NUM>.

The attestation generator <NUM> is configured to receive the verification of the visual appearance and information about the interaction with the rendered element <NUM> as input, and generate, as output, the interaction attestation token <NUM>. Interaction attestation token <NUM> can include a stable ID and/or URI of the declared appearance, a timestamp of the interaction <NUM>, a trust token (e.g., a device integrity token for the operating system or a signed redemption record (SRR) for a web browser), a digital signature generated by the client device <NUM> to verify the integrity of the interaction attestation token (e.g., to verify a trusted device generated the attestation token), XY coordinates of where the interaction occurred within the active window, a client device public key, and an unfalsifiable digital signature created by the operating system or web browser (e.g., to prove the authenticity of the interaction attestation token).

In some implementations, the stable ID and/or URI of the appearance of the rendered element included in the attestation token conveys a real intention of the user's interaction with the rendered element. For example, if the URI points to a "thumbs up" icon for a digital component (e.g., video content), the attestation token attests that the user really clicked on the "thumbs up" button to up-vote the video content. Based on the actual intent attested by the attestation token, the digital content provider <NUM> can act accordingly, e.g. increase the "thumbs up" count.

In some implementations, the user intent corresponds to an action conveyed by the appearance of the rendered element. A report of user interaction can be modified based on the real intention of the user's interaction included in the attestation token. In other words, the report can be modified based on the actual appearance of the rendered element, with which the user believed they interacted. For example, the server can prevent the incrementation of a "vote up" count in response to the rendered element in the attestation token conveying a "vote down" user intention. This can prevent the propagation of false or misleading information and can improve the user's future experience.

In the scenario where the interaction determination module <NUM> and/or the visual analysis module determine that the interaction did not occur at the declared location of the rendered element, the system <NUM> refrains from generating the interaction attestation token <NUM> for the interaction <NUM> with the rendered element <NUM>.

The generated attestation tokens <NUM> can be provided to an interaction validation server <NUM> and stored for further use in aggregation of interactions with rendered elements <NUM> of digital components <NUM>. The interaction validation server <NUM> can include an aggregation engine <NUM> and a response generator <NUM>. Server <NUM> can receive, as input, attestation tokens <NUM>. The aggregation engine <NUM> can aggregate the interaction attestations for multiple interactions with multiple instantiations of the rendered element <NUM>, e.g., multiple validated presentations/interactions with the rendered element <NUM> on multiple client devices <NUM> by multiple users.

In some implementations, the system <NUM> can pair together an interaction attestation token <NUM> for a "finger down" interaction of a user on a touchscreen with an interaction attestation token <NUM> for a "finger up" interaction of the user on the touchscreen, e.g., sequential interactions, and provide the pair of interaction attestation tokens <NUM> to the digital component provider <NUM> or another interested third-party. The two interaction attestation tokens can be paired based in part on having the stable ID and/or URI of the declared appearance of the rendered element, the device public key, and the token creation timestamp included in both attestation tokens to prevent user fraud or misinformation.

The response generator <NUM> can receive interaction attestation tokens <NUM> and false interaction responses and provide, as output, false rendered element responses <NUM>. The false rendered element responses <NUM> can be provided, for example, to the client devices <NUM> and/or digital component providers <NUM> of the digital components <NUM>.

In some implementations, in response to determining that the interaction <NUM> did not occur at the declared location of the rendered element <NUM>, the interaction validation system <NUM> can trigger a false interaction response. A false interaction response can be provided to server <NUM>. Aggregation engine <NUM> can validate the false interaction responses using attestation tokens for the digital component <NUM>, e.g., impression data <NUM> and click conversion attestation <NUM>, as well as interaction attestation tokens <NUM> for the rendered element. The aggregation engine <NUM> can aggregate the validated false interaction responses to maintain user privacy of the users triggering the false interaction responses. A rendered element can be reported as a false rendered element when a threshold portion of users who interact with the rendered element flag it as a false rendered element. Reporting of the false rendered element can be provided to content providers, publishers, and end-users. The distribution of digital components including the false rendered elements can be limited, modified, or halted as result. Digital component provider <NUM> may consider devices <NUM> or application <NUM> as untrustworthy, if they have excessive amounts of false rendered elements. Digital component provider may block or otherwise limit the serving of content to those devices or applications.

In some implementations, the processes performed by the interaction determination module <NUM>, visual analysis module <NUM>, attestation generator <NUM>, and optionally the aggregation engine <NUM> and response generator <NUM> can be performed by more or fewer modules, and each module can include one or more processors.

In some implementations, the methods and systems described with reference to <FIG> in terms of an operating system <NUM> and application <NUM> can be performed, for example, by a web browser. <FIG> is a block diagram of another environment <NUM> in which false rendered element interactions are validated. As depicted in <FIG>, interaction validation system <NUM> is configured to receive a rendering notification <NUM> and an interaction alert <NUM> from a web browser <NUM>. Interaction validation system <NUM> is further configured to receive a declaration <NUM> of a verified appearance of a rendered element <NUM> in an active window <NUM> of a web resource <NUM>. Additionally, interaction validation system <NUM> is configured to receive, from the web browser <NUM>, a screenshot <NUM> of the active window <NUM> displayed of the web resource <NUM>.

As depicted in <FIG>, an interaction <NUM> with a rendered element <NUM> is a user interaction or selection, e.g., using a mouse click, of a "close window" rendered element <NUM> for a digital component <NUM>, e.g., a pop-up window. The rendered element <NUM> depicted in <FIG> can be a false rendered element, where an action associated with the user interaction, e.g., click, with the false rendered element is different than a user intent. For example, the user intent for the interaction <NUM> with rendered element <NUM> is to close the digital component <NUM>, e.g., close the pop-up window. However, instead of closing the digital component <NUM>, the interaction with rendered element <NUM> results in a redirect to a new landing page, or counted as a click on an ad, which is a typical type of ad fraud.

In some implementations, in response to determining that an interaction <NUM> is with a false rendered element, a false interaction response can be triggered. In response to a false interaction response being triggered, the server can intercept a pathway of the client device to prevent the user from reaching a new landing page as a result of the false rendered element. For example, the server can intercept the pathway by preventing the redirection to the new landing page. This reduces the usage of resources and processing in rendering a landing page to which a false rendered element redirects.

<FIG> is a flow diagram that illustrates an example process <NUM> for validating interactions with rendered elements. Process <NUM> for validating interactions with rendered elements can be performed by an operating system <NUM> or another trusted application <NUM> as depicted by <FIG>, and/or can be performed by a web browser <NUM> as depicted by <FIG>.

A rendering notification is received of a rendered element defined in an active window on a client device (<NUM>). A rendering notification <NUM> is received by system <NUM> from an application <NUM> or web resource <NUM>. In some implementations, the operating system <NUM> and/or another trusted application or web browser <NUM> can receive the rendering notification from an application <NUM> or web resource <NUM>, respectively. Rendering notification <NUM> includes information, e.g., a location of the element rendered in the active window, a stable ID and a URI of the rendered element, and a site or domain that called the element, about the rendered element <NUM>, <NUM> that is visible in an active window <NUM> of the client device or website. Rendered element <NUM>, <NUM> can be an element that is presented with a digital component <NUM> on the client device <NUM>.

A declaration of the rendered element defined in the active window on the client device is received (<NUM>). Declaration <NUM> can be received from a repository of declarations <NUM> that are verified by a third-party. Declaration of the rendered element can be requested by the operating system <NUM> or web browser <NUM> in response to receiving the rendering notification of the rendered element being visible in the active window of the client device or webpage.

An interaction with the rendered element at the client device is detected (<NUM>). The interaction can be detected, for example, by the operating system <NUM> of the client device <NUM> or the web browser <NUM>. An interaction <NUM> with the rendered element <NUM> can be, for example, a user's finger touch down on a touchscreen or lift up from the touchscreen on the client device <NUM>. In another example, an interaction <NUM> with a rendered element <NUM> can be a mouse click within the active window <NUM> of the web resource <NUM>.

Based on the received declaration, it is determined whether the interaction occurred at a declared location of the rendered element within the active window (<NUM>). In the embodiment depicted in <FIG>, the operating system <NUM> will examine whether the interaction <NUM>, e.g., click, occurred within an area declared to be occupied by a rendered element. Detecting the interaction <NUM> has occurred at a declared location of the rendered element <NUM> can include receiving, from the operating system <NUM> or other trusted application <NUM>, a selection signal specifying an XY coordinate of a user click or other form of user interaction with the rendered element <NUM> within the active window <NUM>. In some implementations, detecting the interaction <NUM> at the declared location of the rendered element includes receiving a range of X'-X" and a range of Y'-Y" coordinates for the declared area occupied by rendered element in the active window <NUM>, comparing the X coordinate of the interaction <NUM> and the Y coordinate of the interaction <NUM> to the ranges of X'-X" and Y'-Y", and determining whether the X coordinate of the interaction <NUM> and the Y coordinate of the interaction <NUM> are within the ranges of X'-X" and Y'-Y", respectively. In the embodiment depicted in <FIG>, the web browser <NUM> will examine whether the interaction <NUM> occurred within an area declared to be occupied by a rendered element <NUM>.

For a determination that the interaction did occur at the declared location of the rendered element within the active window, a screenshot of the active window on the client device is captured (<NUM>). Before notifying the application <NUM> or web resource <NUM> that the rendered element <NUM>, <NUM> may be malicious, the operating system <NUM> or web browser <NUM> can capture a screenshot <NUM> of the active window <NUM>. The screenshot can be captured, for example, by generating a signal that triggers activation of a screenshot function (e.g., of a camera application installed on the device). The generated signal can be passed to the camera application, which causes the camera application to capture the screenshot of the active window on the client device. In some implementations, the device operating system can perform the screen capture operation directly by copying the display frame buffer. The captured screenshot can then be made available for evaluation by one or more other applications installed on the client device and/or server implemented functions.

A visual appearance of the rendered element in the screenshot is verified with a declared appearance of the rendered element (<NUM>). The operating system <NUM> or web browser <NUM> will compare the screenshot <NUM> against the declared appearance of the rendered element <NUM>, <NUM> from the declaration <NUM>. The comparison can be, for example, a pixel by pixel comparison or another comparison (e.g., block or feature based comparison). If there is a match between the appearance of the rendered element <NUM>, <NUM> in the screenshot <NUM> and the declared appearance of the declaration <NUM> for the rendered element <NUM>,<NUM>, then the operating system <NUM> or web browser <NUM> will generate an interaction attestation <NUM> for the interaction with the rendered element (<NUM>). The interaction attestation token <NUM> can include information about the interaction <NUM>, <NUM>, as described in more detail above with reference to <FIG>.

In some implementations, the application <NUM> or web resource <NUM> that rendered the rendered element <NUM>, <NUM>, respectively, can pair the interaction attestation token <NUM> for a "finger down" with a "finger up" interaction for a same rendered element <NUM>, <NUM> and provide the paired interaction attestation tokens <NUM> to a digital component provider <NUM> or other interested parties for measurement, auditing, or other report-based actions.

For a determination that the interaction did not occur at the declared location of the rendered element within the active window, the system refrains from generating the interaction attestation for the interaction with the rendered element (<NUM>). In some implementations, a false rendered element response is generated by the operating system <NUM> or web browser <NUM> and can be provided to server <NUM> informing the server <NUM> that the application <NUM> or web resource <NUM> that rendered element <NUM>, <NUM>, or that claimed that user had interacted with the rendered element, may be malicious.

<FIG> is block diagram of an example computer system <NUM> that can be used to perform operations described above. The system <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, and an input/output device <NUM>. Each of the components <NUM>, <NUM>, <NUM>, and <NUM> can be interconnected, for example, using a system bus <NUM>. The processor <NUM> is capable of processing instructions for execution within the system <NUM>. In some implementations, the processor <NUM> is a single-threaded processor. In another implementation, the processor <NUM> is a multi-threaded processor. The processor <NUM> is capable of processing instructions stored in the memory <NUM> or on the storage device <NUM>.

A user may be provided with controls allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection of user information (e.g., information about a user's social network, social actions, or activities, profession, a user's preferences, or a user's current location), and if the user is sent personalized content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, the information retention policy, and what information is provided to the user.

Claim 1:
A method performed by data processing apparatus, the method comprising:
receiving a rendering notification (<NUM>) of a rendered element (<NUM>, <NUM>) defined in an active window (<NUM>) on a client device (<NUM>), wherein the rendering notification comprises a location of the rendered element in the active window;
receiving a declaration (<NUM>) of the rendered element defined in the active window on the client device, wherein the declaration includes the location of the rendered element in the active window and a declared appearance of the rendered element;
detecting interaction (<NUM>) with the rendered element at the client device;
determining, based on the received declaration, whether the interaction occurred at the declared location of the rendered element within the active window; and
processing the interaction based on the determination of whether the interaction occurred at the declared location, including:
in response to determining that the interaction occurred at the declared location of the rendered element:
capturing a screenshot (<NUM>) of the active window on the client device;
verifying a visual appearance of the rendered element in the screenshot with the declared appearance of the rendered element; and
generating, based on verifying the visual appearance of the rendered element in the screenshot with the declared appearance of the rendered element, an interaction attestation (<NUM>) for the interaction with the rendered element, thereby validating the interaction; and
in response to determining that the interaction did not occur at the declared location of the rendered element, refraining from generating the interaction attestation for the interaction with the rendered element.