Patent Description:
Content filtering is done for a variety of reasons. For example, content can be filtered to prevent children from being exposed to content that would be inappropriate to view at young ages. Content can also be filtered to prevent "spoilers," i.e., the disclosure of key plot events for television programs or movies. Content can also be filtered out to prevent the harassment of individuals based on gender, ethnicity or religion.

In the context of the Internet, filtering is typically done service side, and is usually application specific. For example, a search engine may have a "safe search" setting enabled that precludes the providing of explicit search results to the user device running the browser. However, the "safe search" setting works only for the search by the search engine, and does not prevent the exposure of content from other websites should a user navigate to the other websites.

By way of another example, an operating system may restrict certain applications from being installed, e.g., applications that are rated "PG-<NUM>" in an application store. However, if the application is not rated, the application may still be installed. <CIT>describes personalized media filtering for a mobile electronic device can be implemented. By supporting receipt of personal media filter criteria, flexible personalization options can be implemented. Personalized media filtering can allow for interactively receiving personal media filter criteria and applying the filter criteria to the media content during the media presentation. One possible blocking response is to block portions of the content. Other possible responses include switching between broadcast stations or playlists. A prevalence metric can indicate how often a particular content item, such as a word, has been filtered from the media content. When the prevalence metric exceeds a threshold, the technology can automatically respond in a variety of ways, such as by blocking a source of the presented media content from future presentation. Interactively establishing personalized filter criteria can be implemented during ongoing presentation, for immediate use in filtering during the presentation. <CIT> describes a multimodal and real-time method for filtering sensitive content, receiving as input a digital video stream, the method including segmenting digital video into video fragments along the video timeline; extracting features containing significant information from the digital video input on sensitive media; reducing the semantic difference between each of the low-level video features, and the high-level sensitive concept; classifying the video fragments, generating a high-level label (positive or negative), with a confidence score for each fragment representation; performing high-level fusion to properly match the possible high-level labels and confidence scores for each fragment; and predicting the sensitive time by combining the labels of the fragments along the video timeline, indicating the moments when the content becomes sensitive. <CIT> describes methods and systems for use in processing encrypted media content through a media processing stack, wherein the media processing stack comprises one or more ordered and successively arranged processing components. These embodiments receive the media content at each successive processing component and pass the media content to a successive processing component; optionally process the media content at each processing component; receive one or more decryption keys associated with the media content at one of the processing components; relay the decryption keys to one or more successive processing components to a decrypting one of the processing components that is capable of decrypting the media content, and decrypt the media content at the decrypting one of the processing components before passing the media content to the successive processing component. <CIT> describes a method, system and product for filtering out data content on a computing device that is integrated into the operating system of the computing device, comprising receiving, by the computing device and in response to a network request by an application, an object associated with content identifiers. The method, system and product further comprise determining, by the computing device and based on an indication of selected filter criteria, a respective weighted probability for the object, and determining, by the computing device, a graphical element for the application based on the weighted probabilities. The method, system and product further comprise outputting, by the computing device and at the graphical user interface of the computing device, the graphical element.

Most existing content filtering solutions apply to specific cases but do not solve the problem for filtering content independent of the web site (in the case of a browser) or other environments (in the case of applications other than browsers). Moreover, restricting the installation of specific applications does not achieve many filtering goals, as the many general purpose applications (e.g., instant messaging applications, e-mail applications) can still be installed and expose the user to undesired content.

In a first aspect, there is provided a computer-implemented method as set forth in claim <NUM>. In a second aspect, there is provided a system as set forth in claim <NUM> and a computer-readable medium as set forth in claim <NUM>. This specification relates to a system and method that analyzes and filters undesired content before the digital content is presented to the user on the user device. More particularly, the systems and methods described below analyze, at a user device, content that is to be presented on the user device and filters the content of undesired items before the content is presented to the user of the user device.

According to the invention, a filtering system operates at an operating system (OS) level on a user device, as opposed to operating at an application level for a specific application. As used in this description, an operation at the OS level means either an operation that has access to higher privileges than an operation at an application level, and/or an operation performed by an operating system, where the operation processes content that is to be displayed. For example, data that is to be rendered on the screen display may be processed to identify content to be filtered. The entire content of the screen may be processed, or only certain content may be processed, e.g., only images may be processed.

By way of a further example, an OS level operation may access data being processed by multiple different applications, and may be an operation that affects the presentation of data by the multiple different applications. Conversely, an application level operation is an operation that may only be able to access data being processed by the application to which it belongs, and cannot affect other applications. Thus, by operating at the OS level, the filtering system is agnostic with respect to the applications for which it can filter content.

One example method includes the following operations. The method accesses, at a user device, data that includes a plurality of content items that are to be presented by an application executing on the user device. The content items may be words, phrases, images, video, and audio. Typically, the content items are being processed by the application for presentation by the application, either in a UI for text, images and video, and over an audio output system for audio.

Prior to the content being presented by the application, a filtering model determines whether the content item is to be presented by the application or filtered by the filtering model. The nature of the determination depends on the content type and the filtering model. For example, text can compared to a list of prohibited words and phrases for a text filter filtering model; conversely, if the filtering model takes into account semantic interpretations of sentences and paragraphs, entire sentences and paragraphs can be filtered, based on the determined semantic interpretation. Any appropriate filtering model can be used, depending on the robustness and sophistication of the filtering desired, and depending on the content type being filtered.

For each content item that is determined to be presented by the application, the filtering model allows the application to present the content item. Conversely, for each content item that is determined to be filtered, the filtering model filters the content item. More particularly, the filtering model precludes, by a system level filtering operation performed at an operating system level and separate from an application level at which the application is executing, presentation of the content item by the application. This can be done, for example, by presenting an opaque mask over filtered text, images and video, or by suppressing or "beeping out" filtered audio.

In some implementations, the filtering element that filters the content item is selectable, and the user may select it to allow the content item to be presented. The filtering model may take this into account and adjust its parameters for future filtering operations. For example, the opaque mask may be selectable, and upon selection, is removed to allow presentation of the filtered content. Likewise, when audio is suppressed, the filtering model may cause the device to state the audio is suppressed, and should the user desire to listen to the audio, what command to use to allow the filtered audio to be presented, e.g., "The song you are about to hear contains explicit lyrics, and I'm going to beep out the explicit words so you can't hear them. However, if you don't find this kind of content objectionable, just say "Let me hear it," and I won't beep out the explicit words.

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of accessing, at a user device, data that includes a plurality of content items that are to be presented by an application executing on the user device; prior to the content being presented by the application: for each content item, determining, at the user device and by a filtering model, whether the content item is to be presented by the application or filtered by the filtering model, for each content item that is determined to be presented by the application, allowing the application to present the content item, and for each content item that is determined to be filtered, precluding, by the filtering model at the user device and by a system level filtering operation performed at an operating system level and separate from an application level at which the application is executing, presentation of the content item by the application. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

By filtering at the OS level, the filtering processes described below may filter for specific content across multiple applications and independent of each application. This reduces the likelihood of filtering leaks that may otherwise occur when filtering is done within specification applications. For example, filtering for pornographic content within a search engine may preclude pornographic content from being provided as search results. However, the same pornographic content may be provided if the user navigates to a site hosting the content, or if the user receives the content by e-mail or instant messaging. Conversely, filtering at the operating system level allows for the user device to implement a filtering process for specific content for all applications that provide content for user consumption.

The analyzing and filtering of content is done independent of the application in which the content is to be presented. This solves the problems related to complexity and resources that accompany separate filtering processes for each application. When filtering is done only at the application level, a separate process must be implemented in each application, and thus each application may consume additional processing resources (e.g., image, video, text and audio processing) when determining whether to filter content. Such implementation of separate processes in each application may not be possible; for example, the device may not have control over processes running on the applications, and providing effective filtering presents challenges.

Moreover, the filtering described below may be done exclusively on the user device, and thus a user's preferences need not be processed at the server level. This reduces the amount of network traffic required when a user changes his or her filtering preferences. For example, search results do not need to be resent when a user desires to change his or her filtering preferences for a particular set of search results. Moreover, filtering being done on the user device protects users' information regarding filtering preferences from leaking out to third parties and being used for other purposes besides filtering (e.g., advertising).

Additionally, on-device filtering reduces resources required for content filtering when such filtering is done server side, as content filtering, such as video filtering, is done only on the devices of the users that desire such filtering.

Another advantage is the ability to filter content that has been received at the device in encrypted form. Because the content is decrypted by an application (e.g., by a chat application), the content is "in the clear" prior to presentation. Thus, circumvention of filtering by encryption is defeated at the device. Moreover, the filtering process need not include an encryption/decryption process, thus simplifying content analysis and filtering.

User devices such as cellular phones, smartphones, PDAs, tablet computers have become very common and an integral part of our daily life. A vast majority of the population now has access to one or more of these devices through which digital content is accessed. These devices are generally capable of performing a variety of tasks such as browsing the Internet, messaging, emailing, downloading content and executing different applications and a variety of other functionalities.

Most of tasks performed by these devices lead to exposure of some sort of digital content to the user. For example, chat applications allow users to send digital content like text messages, images, audio and videos to another user. Browser applications allow access to digital content by means of internet search or digital advertisement. Other examples of such applications may include streaming video applications, online games and cloud based applications.

In all of the above cases where digital content is accessed by applications and presented to the user by the user device, there is always a risk where the user may be exposed to content that is inappropriate for the particular user (e.g., adult content for a child), or that the user may not want to view (e.g., spoilers), or may otherwise be content that is to be filtered from presentation. The systems and methods described below allow a user device to filter such content by filtering at the OS level, across multiple applications simultaneously.

These features and additional features are described in more detail below.

<FIG> is a block diagram of an example user device <NUM> illustrating the functionality of a user device <NUM> while presenting digital content to the user. The user device <NUM> has an application level <NUM> in which applications execute within the user device <NUM> operating system. The operating system operates at an OS level <NUM>, e.g., a kernel mode, or a system level operation, such as a text handler, video handler, or audio handler, while the applications operating at an application level <NUM> other than OS level <NUM>, e.g., a user mode that is less privileged than a root mode, or an operation that occurs within the application environment. Typically, applications operating at the application level <NUM> will only have access to data specific to the application, while processes operating at the OS level <NUM> will have access to data for many applications operating at the application level <NUM>.

As illustrated in <FIG>, a chat application <NUM>, a browser application <NUM>, a music application <NUM> and a video application <NUM> are operating at the application level <NUM>. The user device also includes an I/O system <NUM> that includes a display system <NUM> that renders application data for the application. The I/O system <NUM> also includes a speaker system <NUM> that generates an audio depending on the application that is executing in the application level <NUM>.

The applications executing in the application layer <NUM> generate digital content items for presentation to the user on the user device <NUM>. A content item may be an image, a set of frames of video, text, and each of sequential portions of audio (e.g., one-second increments of audio, for example). When the user is interacting with a particular application through the user device, the digital content of the particular application is directly presented to the user. For example, if the user is interacting with the browser application <NUM>, the browser application <NUM> along with the digital contents of the browser application <NUM> is directly rendered onto the display system <NUM> within the I/O system <NUM> of the user device <NUM>. Similarly, when the user is interacting with the music application <NUM>, the music is directly played using the speaker system <NUM> within the I/O system <NUM>.

A filtering model <NUM> is implemented at the OS level <NUM>. The filtering model <NUM> may receive as input the digital content generated by application executing in the application level <NUM>, processes the input, and identify content to be filtered. Upon identifying the content to be filtered, the filtering model <NUM> precludes the user device <NUM> from presenting the content to the user. How the filtering model identifies content to be filtered, and how the content is actually filtered, is described in more detail below.

The filtering model <NUM> operates at the operating system (OS) level (or simply "system level") on the user device <NUM>, as opposed to operating at an application level for a specific application. As used in this description, an operation at the OS level is an operation that has access to higher privileges than an operation at an application level. For example, an OS level operation may access data being processed by multiple different applications, and may be an operation that affects the presentation of data by the multiple different applications.

The filtering model <NUM> can be implemented as one or more machine-learning model that comprises of multiple parameters wherein the multiple parameters have been trained to identify certain content, such as pornography, expletives, and the like. For example, in some implementations, the filtering model <NUM> can be a collection of various models that can interpret different types of content, such as skin detection model for videos, various object detection models, and word detection models. The filtering model <NUM> may process images, video, text and even audio to detect content to be filtered.

In some implementations, the filtering model <NUM> determines whether content is to be filtered based on user preferences <NUM>. The preferences may specify what type of content, if any, is to be filtered. For example, the filtering model <NUM> may be able to detect multiple types of content - pornography, expletives, gore, etc. The user preferences <NUM>, after being set up by a user, may specify which, if any, of the types of content are to be filtered. Assuming the user desires that only pornography be filtered, then content that is identified as expletives or gore will not be filtered.

In some implementations and depending on the user preferences <NUM>, upon determining that a particular digital content is to be filtered, the particular digital content will not be presented to the user at all. In other implementations, the particular digital content can processed and a modified, a filtered version of the image is presented to the user. For example, the particular digital content can be blurred, pixelated, mutated or otherwise attenuated before presenting it to the user.

These various operations are implemented filtering operations carried out by the filtering model <NUM> at the OS level. For example, the filtering model <NUM> may have access to a library of filtering operations, such as blurring for images and video, muting for audio, and deleting or blurring for text. Images and video that are to be filtered are thus blurred, audio that is to be filtered is muted, and text that is to be filtered is deleted or blurred.

In some implementations, when the filtered content is presented, the user may select the filtered content to reveal the content. For example, in the case of image or text, the user may select the blurred image or text, and the blurred content will be revealed. Alternatively, an administrator role to a specific user may be assigned, and if the user is not the administration, the ability to reveal the filtered content will not be available to the user. In some implementations, the filtering model <NUM> may update itself based on the user's action to view the digital content that was initially filtered.

<FIG> is a block diagram illustrating an example chat application where a sender is transmitting digital content from user device <NUM> to the receiver at the user device <NUM>. The two user devices <NUM> and <NUM> are connected by the network <NUM> enabling transmission of digital content from one device to another. The network <NUM> comprises a local area network (LAN), a wide area network (WAN), the internet or a combination thereof. In some implementations, the chat application and the network <NUM> can enable more than two user devices to communicate over the network wherein multiple user devices can send and receive digital content simultaneously.

The sender uses user device <NUM> to send digital content to the receiver on user device <NUM>. The sender first transmits an image <NUM> of a tree, which is transmitted over the network <NUM>. In some implementations, the chat application can encrypt the image <NUM> such that the chat application on the receiver's user device can decrypt it. Upon receiving the encrypted image <NUM> by the chat application executing on the receiver's user device <NUM>, the chat application decrypts the encrypted image <NUM> to recover the image <NUM>. The chat application on device <NUM> provides the image <NUM> to the filtering model <NUM>. Upon determining by the filtering model <NUM> that the image <NUM> is not to be filtered, the image <NUM> is rendered on the display of the user device <NUM> for presentation to the receiver. More generally, the above example illustrates that any data that is received in encrypted form and then decrypted by an application can be checked for filtering after decryption and before presentation.

The sender again uses the user device <NUM> to send an image <NUM> to the receiver over the network <NUM>. After decryption, the chat application on device <NUM> provides the image <NUM> to the filtering model <NUM>. Upon determining by the filtering model <NUM> that the image <NUM> is to be filtered (e.g., the image depicts content that the user of the user device <NUM> has specified in the user preferences <NUM> to be filtered), the image <NUM> is rendered on the display along with an opaque layer <NUM> on top of the image such that the image is occluded from being presented to the receiver. As described above, in some implementations, the filtered content may be revealed. For example, the opaque layer <NUM>, in some implementations, is selectable and the user may select the opaque layer to see the image <NUM>.

With respect to textual data, the filtering model <NUM> may process text data and detect text in images, such as by optical character recognition. The filtering model <NUM> may include a semantic interpretation model to identify and extract the meaning textual data. Upon receiving the textual data, the semantic interpretation model will utilize semantic detection algorithms to extract the meaning associated with the text. The filtering model <NUM> may then determine whether a semantic interpretation of the content item is a prohibited semantic interpretation listed in a list of prohibited semantic interpretations, for example. If so, then the text may be filtered.

In some implementations, the filtering model <NUM> may further include a sentiment analysis model to determine the sentiment associated with the textual data. In some implementations, these models are based on machine learning algorithms that comprise multiple parameters adjusted to learn rules of natural language by training on a set of examples like a large corpus or a collection of sentences and perform statistical inference. Upon performing textual analysis and after extracting the meaning of the textual data, the filtering system determines whether the textual data is to be filtered, based on user preferences. For example, textual content related to sex or regulated substances may be filtered, based on user preferences.

In some implementations, the model <NUM> may be trained to identify sections of text to filter. The sections may be identified by content headers or topic identifiers. For example, an article may include a header "Spoiler Alert" that indicate the text following the header includes spoilers for a movie. The model <NUM> may be trained to filter all text in the article following the "Spoiler Alert" header, for example. Other text filters may include filtering based on word matches to prohibited words.

With respect to audio, the filtering model <NUM> may process the audio to detect speech and other types of content (e.g., different styles of music, such as classical, country, jazz, metal, and the like). For example, the music application <NUM> executing within the application level <NUM> may play audio songs using the speaker system <NUM>. A user may desire the explicit lyrics be filtered. In such implementations, the filtering model <NUM> will process the audio data to detect for explicit words. Upon determining that particular words in a song are explicit, and based on the user preferences, the filtering system <NUM> may filter out the words, such as by suppressing the audio or adding a censor beep sound effect in place of the word while playing the music on the user device <NUM>.

In still other implementations, the content may be tagged with descriptors that describe the content, and the model <NUM> may determine the content based on the tags.

<FIG> is the flow diagram of an example process <NUM> of filtering digital content generated by an application executing in the application layer. The process <NUM> is implemented in a computer system, such as a user device <NUM> that implements the model <NUM>.

The process <NUM> accesses digital content generated by the applications executing in the application level for presentation to the user (<NUM>). The digital content generated by the applications is received as input by the content filtering system <NUM>. For example, the chat application <NUM> executing in the application level <NUM> receives a text that has to be presented to the user of the user device. The chat application <NUM> generates digital content comprising an application specific user interface with the received text for presentation to the user. In some implementations, instead of providing the digital content to the I/O system <NUM> of the user device <NUM>, the chat application <NUM> (and other applications at the application level) provide the digital content as input to the filtering model <NUM>. In other implementations, the filtering model <NUM> may instead receive the digital content from a system level I/O process. For example, when the model <NUM> is installed on a user device, the OS may provide the digital content to the model in a step prior to video rendering or audio output. In this way, the model <NUM> captures, from the application, the content that has been processed by the application for presentation by the application. The model <NUM> may then process the digital content and then, based on user preferences and the type of content detected, cause the user device to perform one or more filtering operations.

After receiving the digital content as input by the filtering model <NUM>, the process <NUM> determines whether the digital content is to be presented or filtered (<NUM>). If the process <NUM> determines the content is to be presented, then the process presents the digital content to the user (<NUM>). For example, upon determining that the digital content to be presented in a user interface in application <NUM> is not to be filtered, the digital content is provided as input to the I/O system <NUM>. The I/O system <NUM> then renders the digital content onto the display system <NUM> for presentation to the user.

If the digital content is determined to be filtered, the process <NUM> presents a selectable opaque layer over the digital content or a portion of the digital content that was determined to be inappropriate (<NUM>). For example, after receiving the image <NUM> by the user device <NUM>, and upon determining that the image <NUM> is to be filtered, the filtering system <NUM> presents a selectable opaque layer <NUM> over the image for presentation thereby occluding the inappropriate content. Other methods of filtering include not presenting the content at all or blurring, pixelating, mutilating or otherwise attenuate the content before presenting it to the user.

The user, upon being presented with an opaque layer over the filtered content, may wish to see the filtered content. Upon selecting the opaque layer, the user may be provided with an option to view the unaltered digital content. Based on user's actions, the filtering system may update the user preferences (<NUM>). In some implementations, user preferences can be independent of user feedback where the user may specify filtering parameters that will not change based on user actions.

In situations in which the systems discussed here collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether applications or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location), or to control whether and/or how to receive content that may be more relevant to the user. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and used by a content server.

Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus.

The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., a FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's user device in response to requests received from the web browser.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components.

The computing system can include users and servers. A user and server are generally remote from each other and typically interact through a communication network. The relationship of user and server arises by virtue of computer programs running on the respective computers and having a user-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a user device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the user device). Data generated at the user device (e.g., a result of the user interaction) can be received from the user device at the server.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any features or of what may be claimed, but rather as descriptions of features specific to particular embodiments.

Claim 1:
A computer-implemented method, comprising:
accessing (<NUM>), at a user device (<NUM>), digital content data that includes a plurality of content items generated by applications (<NUM>, <NUM>, <NUM>, <NUM>) executing at an application level (<NUM>) on the user device for presentation by an input/output system (<NUM>) of the user device, wherein the accessing comprises capturing, from the applications, the digital content data including the content items that have been processed by the applications for presentation by the applications;
prior to the captured digital content data being presented to the user by the input/output system:
for each content item of the captured digital content data, determining (<NUM>), at the user device and by a filtering machine learning, ML, model (<NUM>) of a system level filtering operation performed at the operating system level (<NUM>) and separate from the application level at which the applications are executing, whether the content item is to be presented to the user by the input/output system or filtered by the filtering ML model;
for each content item of the captured digital content data that is determined to be presented by the input/output system to the user, allowing (<NUM>) the input/output system to present the content item; and
for each content item of the captured digital content data that is determined to be filtered, precluding (<NUM>), by the filtering ML model at the user device and by the system level filtering operation performed at the operating system level and separate from an application level at which applications are executing, presentation of the content item to the user by the input/output system.