Deep links for native applications

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for automatically generating native application deep links and indexing content retrieved from the deep links.

BACKGROUND

The Internet provides access to a wide variety of information. For example, digital image files, video and/or audio files, as well as web page resources for particular subjects or particular news articles, are accessible over the Internet. With respect to web page resources, many of these resources are designed to facilitate the performing of particular functions, such as banking, booking hotel reservations, shopping, etc., or to provide structured information, such as on-line encyclopedias, movie databases, etc.

A variety of search engines are available for identifying particular web page resources accessible over the Internet. With the advent of tablet computers and smart phones, native applications that facilitate the performance of the same functions facilitated by the use of web page resources are now being provided in large numbers.

A user's informational need may thus be satisfied by providing search results that identify either one (or both) of a particular web page resource or a native applications that facilitates the performance of the same functions facilitated by web page resource, or presents the same or very similar information as the web page resource.

SUMMARY

The present disclosure relates to native application deep linking, and more specifically to systems and methods for automatically generating native application deep links.

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving publisher affiliation data for native applications that define, for each native application, an affiliated publisher for the native application; for each native application of a set of native applications, determining, based on the publisher affiliation data, whether it is affiliated with a publisher that provides content addressed by URIs determined for the native application; for only the native applications that are determined to be affiliated with a publisher that provides content addressed by URIs determined for the native application: selecting URIs based on the URI pattern for the native application, indexing content accessible by the URI for the native application in an index that is searchable by a search engine. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

Another innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of determining a uniform resource identifier (URI) pattern for a native application; instantiating the native application and selecting a first URI based on the URI pattern to generate an application page in the native application display environment; indexing application page data of the application page in an index that is searchable by a search engine; iteratively processing linked application pages for the native application until a cessation event occurs, the iterative processing comprising for each iteration; determining, from the application page, outbound URIs including in the first application page; selecting one or more of the outbound URIs to generate one or more subsequent application pages in the native application display environment; indexing the application page data for each of the one or more subsequent application pages in the index. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. The generation of the deep links makes it easier for application developers to participate in native application indexing, which in turn helps drive usage and re-engagement of their applications. The system also lowers the barrier for participation.

DETAILED DESCRIPTION

A system provides native application search results. As used herein, a native application operates independent of a browser application on the user device. A native application is an application specifically designed to run on a particular user device operating system and machine firmware and not within a browser. Native applications thus differ from browsers, browser-based applications and browser-rendered resources. The latter require all, or at least some, elements or instructions downloaded from a web server each time they are instantiated or rendered. Furthermore, browser-based applications and browser-rendered resources can be processed by all web-capable mobile devices within the browser and thus are not operating system specific as are native applications. Furthermore, as used herein, a native application search result is a search result that corresponds to a particular native application and that, when selected, invokes the particular native application.

Examples of native application include applications to create text documents, edit photographs, play music, interface with a remote banking system, and so on. Mobile native applications are designed to operate on mobile devices such as smart phones, tablets, and so on. Web native applications are designed to run within (both visually and functionally) web browser software. At a very high level, native applications, or simply “apps,” provide a user with access to content and/or functionality.

Internet search engines provide the ability to search an indexed collection of web pages that exist on the Internet. The typical search engine provides, in response to a query, a list of results relevant to the query, often with hyperlinks allowing a user to quickly navigate to a result of interest. A user may select a linked result and thereby be provided with a web page(s) containing content about a celebrity, product, business, etc. The links may be to a site's “home page” (e.g., a welcome and search page for a site aggregating cooking information such as recipes) or may be a “content page” that a user can access when navigating from a site's home page (e.g., a specific recipe at the cooking information site). Links to content pages, as opposed to home pages, are referred to as “deep links” (and the process for crawling, indexing, and serving these content pages is similarly referred to as “deep linking”).

The ability to navigate to a page, as well as the accessibility of content and functionality that that page provides, is dictated in part by the nature of the published page. For example, certain pages may contain dynamic content (e.g., asynchronous java script and XML (AJAX), Adobe Flash, etc.) that are difficult or impossible to crawl and retrieve. Consequently, these pages may not be available in response to a search, a user may not be able to save the page as a bookmark or the like, and page navigation via browser controls (e.g., forward and back in history) may not operate as intended.

Some native applications lack some of the basic properties of websites when it comes to linking First, when a search engine crawls a native application, it cannot simply inspect the existing link to a page and look at the outbound links of the page like it can by looking at the HTML of a webpage. Second, many native applications do not link to one another the way that websites inherently do, so search engines cannot rely on crawling the web to discover app deep links like they can for discovering web links. This makes it challenging for native application search engines in that it is much more difficult to discover links that a native application supports. If a native application search engine cannot discover links, then it cannot index them, retrieve them, and serve them to users.

Furthermore, some native applications have corresponding web pages for the URIs the native applications access, while others do not. Of those that do have corresponding web pages, some native applications maintain duplicate or corresponding content and functionality between the native application and web platforms, while other native applications have small or large degree differences between mobile and web platforms.

Some systems for indexing native applications rely on developers to publish their native application deep links either as HTML markup on their webpages or in their existing sitemaps. That is, the native application provider must have both a web platform and at least map the native application pages at the website. Some services providing native application deep linking also rely on developers publishing their native application deep links on their webpages.

According to the present disclosure, native application deep links may be automatically generated for indexing purposes. There are numerous methods for native application deep link generation, depending on the nature of the native application deep links. For example, some native applications support deep links having URIs that conform to web-based uniform resource locators (URLs), such as http://www.example.com, etc. Other native applications use a custom URI that does not necessarily conform to a particular HTTP protocol.

For native applications that support deep linking based on web-based URLs, a process to generate native application deep links includes:1) Native application developers verify their official website with a centralized indexing service. The service then indexes the native application based on the previously indexed web URLs discovered through web crawling.2) A native application data crawling and indexing system inspects registration information, such as a manifest file, of the native application to determine if it supports the appropriate URL structure of the website.3) If the native application does support the appropriate URL structure, the native application data crawling and indexing system can determine and prioritize links based on criteria such as popularity. The native application data crawling and indexing system will also respect any non-index tag in the registration information which specifies which URL paths should not be indexed.

According to a second aspect of the present disclosure, for native applications that support custom deep linking, a system implements a method comprising the following steps:1) Native application developers implement a native application indexing API that allows specifying the URI address of a particular native application document, its corresponding web URL (if any), and all outbound app deep links and their associated web URLs (if any) that are on the document.2) Once a native application is updated at a digital distribution source, the native application data crawling and indexing system inspects the native application to determine if it is using the native application indexing API. If so, then the native application data crawling and indexing system launches the native application and looks for the link to native applications “home page.” The native application data crawling and indexing system also looks for any outbound URI links on the application page. If there are outbound URIs, then the native application data crawling and indexing system will index each of those links (or only affiliated links), and will also in turn process the subsequent native application pages for outbound URIs as well. Thus the native application data crawling and indexing system is able to automatically discover links. And at each step, the native application data crawling and indexing system can check the current link of the application page against the source link as a method of authentication.3) The native application data crawling and indexing system can be used to maintain a list of those links that are actually being viewed by the user within a native application, and provide that list to the native application data crawling and indexing system. The native application data crawling and indexing system uses this list as an additional source of links, in addition to the automatic discovery described in the previous step. Furthermore, the lists of links that are actually viewed can be used to determine link popularity, and the native application data crawling and indexing system can then prioritize crawling based on the popularity of links.

Accordingly, the systems and methods described herein can provide an automatic indexing of native application deep links. The systems and methods may further determine and utilize link prioritizations from actual user activity. Prioritization permits, inter alia, optimizing the utilization of search resources. Finally, systems and methods are disclosed that can accommodate apps with and without corresponding websites, overcoming the limitations of existing techniques that require apps to publish links through corresponding webpages.

FIG. 1is a block diagram of an example environment100in which deep links for native applications are generated. A computer network102, such as the Internet, connects resource publisher web sites104, application publishers106, user devices108and a search engine120.

A resource publisher website104includes one or more web resources105associated with a domain and hosted by one or more servers in one or more locations. Generally, a resource publisher website is a collection of web pages formatted in hypertext markup language (HTML) that can contain text, images, multimedia content, and programming elements. Each website104is maintained by a content publisher, which is an entity that controls, manages and/or owns the website104.

A web page resource is any data that can be provided by a publisher website104over the network102and that has a resource address, e.g., a uniform resource locator (URL). Web resources may be HTML pages, images files, video files, audio files, and feed sources, to name just a few. The resources may include embedded information, e.g., meta information and hyperlinks, and/or embedded instructions, e.g., client-side scripts.

An application publisher website106may also include one or more web resources105, and also provides native applications107. A native application107is an application specifically designed to run on a particular user device operating system and machine firmware. Native applications107may include multiple versions designed to run on different platforms. For example, native applications corresponding to a movie database website may include a first native application that runs on a first type of smart phone, a second native application that runs on a second type of smart phone, a third native application that runs on a first type of tablet, etc.

An application page is a particular display environment within a native application and in which is displayed content, such as text, images, and the like. An application page is specific to the particular native application, and the native application is specific to the particular operating system of the user device108. An application page differs from a rendered web resource in that the application page is generated within and specific to the native application, while a web resource may be rendered in any browser for which the web page resource is compatible, and is independent of the operating system of the user device.

Some publishers104and106may be the same, and provide the same content in both web resources and native applications. The management of the delivery of such content is handled by a publisher backend130. The publisher backend130includes a publisher store132that stores content the publisher provides, a web server134that provides the content from the publisher store as web resources (e.g., web pages), and a native application server136that handles requests from a native application. The web pages correspond to native application pages, and thus for many native application URIs there are corresponding web page URIs. Some native applications may also simply use the same URLs as the web pages to which they correspond. The content of the publisher store132is accessible for presentation on both the web resources and the corresponding native application pages. Accordingly, such content is referred to as “synchronized” content.

Examples of such publishers are news publishers, which may have a “mobile news app” for reading news content on a mobile device. The news content (e.g., the text of news story) provided on web resources105are the same content that is provided by the corresponding application pages of the native application107.

A user device108is an electronic device that is capable of requesting and receiving web page resources105and native applications107over the network102. Example user devices108include personal computers, mobile communication devices, and tablet computers.

The web index116is an index of publisher content that has, for example, been built from crawling the publisher websites104, by receiving data feeds from the publisher websites104, or by other appropriate methods of collecting and indexing data.

The native application index114stores data relating to native applications107. The application index114stores, for example, a list of native applications provided by the publishers104and identifiers that identify the native applications. Furthermore, in some implementations, publishers104may specify that certain native applications107are used to access and display synchronized data, and this information may be stored in the application index. For example, a news publisher may specify that a news reading application it provides displays synchronized content that is displayed on the news publisher's website.

The user devices108submit search queries to the search engine110. In response to each query, the search engine110accesses the web index116and the application index114to identify content that is relevant to the query. The search engine110may, for example, identify the resources and applications in the form of web resource search results and native application search results, respectively, by of use a search result generator116. Once generated, the search results are provided to the user device108from which the query was received.

A web resource search result is data generated by the search engine110that identifies a web resource, based on the content of the resource that satisfies a particular search query. A web resource search result for a resource can include a web page title, a snippet of text extracted from the resource, and a uniform resource identifier (URI) for the resource, e.g., the uniform resource locator (URL) of the web page. When selected at a user device, the web resource search result causes the user device to generate a request for the resource located at the URL. The web resource that is received is then displayed in a browser application.

A native application search result specifies a native application and is generated in response to a search of the application index114and the web index116, as described in more detail below. When selected at a user device, the native application search result causes the native application installed on the user device to request the synchronized content. Once the native application receives the requested content, the native application displays the content in a user interface of the native application.

To generate the application index114, the search system utilizes an application crawling and indexing system120.FIG. 2is a block diagram of an application crawling and indexing system120. Operation of the system120in the context of native applications that use URLs is described with reference toFIG. 3. A variation of the operation for native applications that use custom URIs is described with reference toFIG. 4.

FIG. 3is a flow diagram of an example process300for generating deep links for native applications. The process300is implemented in a data processing apparatus of one or more computers.

The process300receives at the data collector204publisher affiliation data202for native applications that define, for each native application, an affiliated publisher for the native application (302). For example, publishers provide data that a particular native application is affiliated with the publisher, such as an association of a native application identifier with a site identifier of the publisher. One example site identifier is a domain name, but other types of data for defining an affiliation between a publisher and native application can be used.

The process300, for a native application, determines based on the publisher affiliation data whether the native application is affiliated with a publisher that provides content addressed by URIs determined for the native application (304). For example, the application package crawler206extracts from an application manifest file109(or other data defining similar application specifications) a scheme, host and path of a URI format for the native application. For example, in the case of an Android manifest, the following is extracted from an intent filter section:<data android:scheme=“http”android:host=“example.com”android:pathPrefix=“/gizmos”/>

Here the scheme is “http,”, the host is “example.com,” and the path is “/gizmos.” The process determining a URI pattern for the native application, and whether the URI pattern defines an affiliated publisher for the native application that matches an affiliated publisher for the native application specified by the publisher affiliation data202. To illustrate, for the example data above, if the affiliation data202defines a scheme of “http” for a native application, and a host of “example.com,” and the manifest109of the native application107defines the same scheme and host, then the native application is affiliated with the host.

If the determination is positive, the URI scheme generator208stores the data for the native application in the URI scheme table210, and then the process300selects URIs based on the URI pattern for the native application (306). For example, in some implementations, the URI selector212retrieves the URI pattern from the URI scheme table210. The selector212then searches the web index116for URL's that that include the URI pattern. In the example above, candidate URLs that being with http://example.com/gizmos/ will be processed by the selector212. The selector212may discard URLs that may include an indication they are not to be crawled; that have been recently searched and indexed for the native application, that are indicated as being deleted or inactive; or URLs that meet some other exclusion criteria.

The remaining URLs that are selected are provided to a URI processor214, which the collects data from content accessible by the URL. The process300then indexes content accessible by the URL for the native application in an index that is searchable by a search engine (308). The collection and indexing of data can be done by any appropriate process. In one example implementation, system120instantiates a virtual machine emulating an operating system for a user device. The virtual machine may, in some implementations, be a modified version of the operating system and includes extractors that extract data from application pages as described in more detail below.

The system120also instantiates, within the virtual machine, a native application107that generates application pages for display on a user device within the native application107, and then accesses, within the virtual machine, application pages of the native application generated in response to processing the selected URLs. For each application page, the system110generates application page data describing content of the application page. The content of the page may include, for example, text displayed on the application page; images displayed on the application page; links on the application page to other application pages or other web resources; and other content that is appropriate for indexing.

In some implementations, the virtual machine includes extractors that extract content data for indexing. The extracted content data is, for example, data that is provided to a rendering process of the native application. The rendering process renders content based on the data for display on a user device. Use of extractors allows for more accurate identification of the various content of an application page. For example, the text extractor extracts text data provided to a rendering process of the native application. The text data specifies the text that is to be rendered in the application page. Thus, rather than processing an image of the application page, or processing binary data of the display, the virtual machine receives the actual text that is to be rendered in the environment of the native application107.

Other extractors can likewise be used, such as an image extractor and a list extractor. The image extractor provides image data of an image that is to be rendered in the environment of the native application107, and the list extractor provides list data of a list of scrollable items that is rendered in the environment of the native application107. Other data can also be extracted, such as application page link data describing links within the application page that link to another application page; web page link data describing links within the application page that link to a web resource referenced by a uniform resource locator and that, when selected, instantiate a browser application that renders the resource in a browser environment separate from the native application; etc.

The extractors described above, and other appropriate data extractors, can be implemented using appropriate data handlers for a particular operating system. For example, for the Android™ operating system, the extractors may be implemented using TextView objects, ImageView objects, and ListView objects, respectively. The virtual machine processes the objects to extract the corresponding data, e.g., by including instructions that cause the virtual machine to store for indexing the corresponding data that is provided for rendering.

Returning to304, if the determination is negative, then the process300does not process URIs of publisher (310). This facilitates the indexing of publisher content for only the native application that are affiliated with the publisher (e.g., published by the publisher or approved by the publisher to display publisher content).

For native applications that support custom deep linking that does not necessarily conform to a website, a different process is used to generate deep linksFIG. 4is a flow diagram of another example process for generating deep links for native applications. The process400is implemented in a data processing apparatus of one or more computers.

The process400determines a URI pattern for a native application (402). For example, the application manifest109may be accessed to determine the URI scheme. The URI scheme may be used to define a “top level” native application URI resource, or may be used to select URIs that have already been indexed for the native application.

The process400instantiate the native application and selects a first URI based on the URI pattern to generate an application page in the native application display environment (404). For example, a virtual machine is instantiated, and on the virtual machine the native application is also instantiated. The URI is provided to the native application and the application page is rendered.

The process400begins an iterative discovery process (406). The process will continue until a complete set of URIs have been processed.

The process400determines, from the application page, outbound URIs including in the first application page (408). For example, a native application API determines from the application page data URIs that link to other native application URIs. The determination can be made, for example, by processing text extracted from the application page for matches to the URI pattern generated from the manifest109; or based on the API selecting mark-up text that defines outgoing URIs.

The process400selects one or more of the outbound URIs to generate one or more subsequent application pages in the native application display environment (410). In some implementations, all outgoing URIs are selected. In other implementations, only the outbound URIs conforming to the URI pattern of the native application are selected.

The process400indexes the application page data for each of the one or more subsequent application pages in the index (412). As described above, the system110can use a virtual machine, or some other construct, to extract the application page data for indexing.

The process400determines if the iterative discovery process has ended (414). If the iterative discovery process has ended, then the process400ends application indexing for the native application (416). Otherwise, the process400returns to step406and begins another iteration.

In some implementations, when actual users view content within the native applications, the native application API can be used to let the application crawling and indexing system120know that links that are actually being viewed by users. The application crawling and indexing system120can use this information as an additional source of links, in addition to the organic discovery described above. Furthermore, the application crawling and indexing system120can also prioritize crawling based on the popularity of links.