Contextual Alternate Text for Images

In one embodiment, one or more server computing devices receive from a client computing device associate with a user a request for a structured document including an image; the server computing devices access a data store for data associated with the image; the server computing devices construct based at least in part on at least some of the data associated with the image a text string corresponding to the image in the structured document; the text string is configured to be audibly read out by the client computing device; the server computing devices also send the structured document with the text string to the client computing device for presentation to the user.

DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG. 1illustrates an example network environment100associated with a social-networking system. Network environment100includes a user101, a client system130, a social-networking system160, and a third-party system170connected to each other by a network110. AlthoughFIG. 1illustrates a particular arrangement of user101, client system130, social-networking system160, third-party system170, and network110, this disclosure contemplates any suitable arrangement of user101, client system130, social-networking system160, third-party system170, and network110. As an example and not by way of limitation, two or more of client system130, social-networking system160, and third-party system170may be connected to each other directly, bypassing network110. As another example, two or more of client system130, social-networking system160, and third-party system170may be physically or logically co-located with each other in whole or in part. Moreover, althoughFIG. 1illustrates a particular number of users101, client systems130, social-networking systems160, third-party systems170, and networks110, this disclosure contemplates any suitable number of users101, client systems130, social-networking systems160, third-party systems170, and networks110. As an example and not by way of limitation, network environment100may include multiple users101, client system130, social-networking systems160, third-party systems170, and networks110.

In particular embodiments, user101may be an individual (human user), an entity (e.g., an enterprise, business, or third-party application), or a group (e.g., of individuals or entities) that interacts or communicates with or over social-networking system160. In particular embodiments, social-networking system160may be a network-addressable computing system hosting an online social network. Social-networking system160may generate, store, receive, and send social-networking data, such as, for example, user-profile data, concept-profile data, social-graph information, or other suitable data related to the online social network. Social-networking system160may be accessed by the other components of network environment100either directly or via network110. In particular embodiments, social-networking system160may include an authorization server (or other suitable component(s)) that allows users101to opt in to or opt out of having their actions logged by social-networking system160or shared with other systems (e.g., third-party systems170), for example, by setting appropriate privacy settings. A privacy setting of a user may determine what information associated with the user may be logged, how information associated with the user may be logged, when information associated with the user may be logged, who may log information associated with the user, whom information associated with the user may be shared with, and for what purposes information associated with the user may be logged or shared. Authorization servers may be used to enforce one or more privacy settings of the users of social-networking system30through blocking, data hashing, anonymization, or other suitable techniques as appropriate. In particular embodiments, third-party system170may be a network-addressable computing system that can host websites or application. Third-party system170may generate, store, receive, and send third-party system data, such as, for example, web pages, text, images, video, audio, or applications. Third-party system170may be accessed by the other components of network environment100either directly or via network110. In particular embodiments, one or more users101may use one or more client systems130to access, send data to, and receive data from social-networking system160or third-party system170. Client system130may access social-networking system160or third-party system170directly, via network110, or via a third-party system. As an example and not by way of limitation, client system130may access third-party system170via social-networking system160. Client system130may be any suitable computing device, such as, for example, a personal computer, a laptop computer, a cellular telephone, a smartphone, or a tablet computer.

FIG. 2illustrates example social graph200. In particular embodiments, social-networking system160may store one or more social graphs200in one or more data stores. In particular embodiments, social graph200may include multiple nodes—which may include multiple user nodes202or multiple concept nodes204—and multiple edges206connecting the nodes. Example social graph200illustrated inFIG. 2is shown, for didactic purposes, in a two-dimensional visual map representation. In particular embodiments, a social-networking system160, client system130, or third-party system170may access social graph200and related social-graph information for suitable applications. The nodes and edges of social graph200may be stored as data objects, for example, in a data store (such as a social-graph database). Such a data store may include one or more searchable or queryable indexes of nodes or edges of social graph200.

In particular embodiments, a concept node204may represent a third-party webpage or resource hosted by a third-party system170. The third-party webpage or resource may include, among other elements, content, a selectable or other icon, or other inter-actable object (which may be implemented, for example, in JavaScript, AJAX, or PHP codes) representing an action or activity. As an example and not by way of limitation, a third-party webpage may include a selectable icon such as “like,” “check in,” “eat,” “recommend,” or another suitable action or activity. A user viewing the third-party webpage may perform an action by selecting one of the icons (e.g., “eat”), causing a client system130to send to social-networking system160a message indicating the user's action. In response to the message, social-networking system160may create an edge (e.g., an “eat” edge) between a user node202corresponding to the user and a concept node204corresponding to the third-party webpage or resource and store edge206in one or more data stores.

In particular embodiments, a pair of nodes in social graph200may be connected to each other by one or more edges206. An edge206connecting a pair of nodes may represent a relationship between the pair of nodes. In particular embodiments, an edge206may include or represent one or more data objects or attributes corresponding to the relationship between a pair of nodes. As an example and not by way of limitation, a first user may indicate that a second user is a “friend” of the first user. In response to this indication, social-networking system160may send a “friend request” to the second user. If the second user confirms the “friend request,” social-networking system160may create an edge206connecting the first user's user node202to the second user's user node202in social graph200and store edge206as social-graph information in one or more of data stores164. In the example ofFIG. 2, social graph200includes an edge206indicating a friend relation between user nodes202of user “A” and user “B” and an edge indicating a friend relation between user nodes202of user “C” and user “B.” Although this disclosure describes or illustrates particular edges206with particular attributes connecting particular user nodes202, this disclosure contemplates any suitable edges206with any suitable attributes connecting user nodes202. As an example and not by way of limitation, an edge206may represent a friendship, family relationship, business or employment relationship, fan relationship, follower relationship, visitor relationship, subscriber relationship, superior/subordinate relationship, reciprocal relationship, non-reciprocal relationship, another suitable type of relationship, or two or more such relationships. Moreover, although this disclosure generally describes nodes as being connected, this disclosure also describes users or concepts as being connected. Herein, references to users or concepts being connected may, where appropriate, refer to the nodes corresponding to those users or concepts being connected in social graph200by one or more edges206.

In particular embodiments, social-networking system160may create an edge206between a user node202and a concept node204in social graph200. As an example and not by way of limitation, a user viewing a concept-profile page (such as, for example, by using a web browser or a special-purpose application hosted by the user's client system130) may indicate that he or she likes the concept represented by the concept node204by clicking or selecting a “Like” icon, which may cause the user's client system130to send to social-networking system160a message indicating the user's liking of the concept associated with the concept-profile page. In response to the message, social-networking system160may create an edge206between user node202associated with the user and concept node204, as illustrated by “like” edge206between the user and concept node204. In particular embodiments, social-networking system160may store an edge206in one or more data stores. In particular embodiments, an edge206may be automatically formed by social-networking system160in response to a particular user action. As an example and not by way of limitation, if a first user uploads a picture, watches a movie, or listens to a song, an edge206may be formed between user node202corresponding to the first user and concept nodes204corresponding to those concepts. Although this disclosure describes forming particular edges206in particular manners, this disclosure contemplates forming any suitable edges206in any suitable manner.

In addition, the degree of separation between any two nodes is defined as the minimum number of hops (or edges) required to traverse the social graph from one node to the other. A degree of separation between two nodes can be considered a measure of relatedness between the users or the concepts represented by the two nodes in the social graph.

A structured document such as a web page may include, for example, page layout information, scripts, page content such as text (e.g., ASCII or HTML), media data (e.g., graphics, photos, video clips), and executable code objects (e.g., a game executable within a browser window or frame). Structured documents may be implemented with languages and technologies such as Hypertext Markup Language (HTML), Extensible Markup Language (XML), Extensible Hypertext Markup Language (XHTML), JavaScript, WebGL, Cascading Style Sheet (CSS) including CSS animations and transitions, and, frequently, Java. A structured document may itself include references to multiple structured documents and contents. For example, a web page may include one or more inline references by incorporating Uniform Resource Locations (URLs) and/or script code (e.g., JavaScript, PHP, AJAX) that in response to a user event (e.g., a mouse click, a mouse hover-over), causes an application displaying the web page in a graphical user interface to dynamically retrieve content specified by an URL and the script code.

A structured document such as a web page may include an image using an HTML <img> tag. Meanwhile, an <img> tag may also include an alternate text using the alt attribute of the <img> tag. For example, an image of Golden Gate Bridge with a corresponding URL “www.example.com/picture.jpg” and an alternate text “Picture of Golden Gate Bridge” may be constructed as <img src=“www.example.com/picture.jpg” alt=“Picture of Golden Gate Bridge”> in the web page. A client computing device displaying the web page may display the alternate text “Picture of Golden Gate Bridge” if the image is not available for displaying (e.g., if a server hosting the image's URL is not responding).

A client computing device may display a web page with an image, while audibly reading out text content of the web page and the alternate text for the image (e.g., using a text-to-speech software program and a speaker of the client computing device). Reading out audibly alternate texts of images in a web page and other text content of the web page is useful for accessibility of a user who is visually impaired or visually disabled (e.g., a blind person). For example, a client computing device displaying the web page with the image of Golden Gate Bridge described above may audibly read out text content from the beginning of the web page. While reaching the image, the client computing device may audibly read out the alternate text “Picture of Golden Gate Bridge.” In this way, a blind person using the client computing device can know the content of the web page fully. Without the alternate text, a blind person using the client computing device displaying the web page may only hear text content before (and after) the image in the web page, and an audible read-out “link”, indicating an existence of a URL link (of the image) between the text content.

Ordinarily, an alternate text for an image in a web page is static. For example, a website hosting an image may create an alternate text of the image. The alternate text may be the same for the life time of the image. The web page with the image may directly include the image's alternate text, regardless of other content of the web page or viewers of the web page. Particular embodiments describe methods for dynamically constructing an alternate text of an image included in a web page. Particular embodiments may dynamically constructing an alternate text of an image included in a web page based on contextual information, such as what other content is included in the web page, or who is a viewer of the web page.

FIG. 3illustrates an example method300for dynamically constructing an alternate text of an image included in a structured document. Particular embodiments may be implemented by a computing process (e.g., of an application) executed by one or more servers serving the structured document. Particular embodiments may be implemented by a computing process executed by one or more server computing devices of a social-networking system as described earlier.

The method300may begin at step310. In particular embodiments, at step310, one or more servers of the social-networking system may receive from a client computing device a request for a structured document comprising an image. The client computing device may be a desktop computer, a laptop computer, a tablet computer, a smartphone, or any suitable client computing device that is configured to display the structured document (e.g., a web page). The client computing device may be associated with a particular user (e.g., a viewer of the structured document when it's served by the servers). The particular user may be a user of the social-networking system. The image may comprise a photograph or any suitable image file in any suitable format (e.g., Graphics Interchange Format or GIF format, Joint Photographic Experts Group or JPEG format, Exchangeable image file format or Exif format). The image may be stored in a data store of the social-networking system or a third-party system.

In particular embodiments, the image may correspond to a concept node of a social graph of the social-networking system. The concept node corresponding to the image may also comprise additional information associated with the image. For example, the concept node may include an author (or a source) of the image. The concept node may include a location (e.g., a name of a place, a pair of geographical coordinates, a zip code, and so on) of the image. The concept node may include a time stamp of the image. Particular embodiments contemplate any suitable information that may be included in the concept node corresponding to the image. The social-networking system may store the concept node corresponding to the image in one or more data stores of the social-networking system.

In addition, the social graph may comprise one or more edges connecting the concept node (corresponding to the image) to other user or concept nodes of the social graph. For example, the social graph may include an edge connecting the concept node to a user node, while the edge representing a user corresponding to the user node is tagged in the image corresponding to the concept node. The social graph may include an edge connecting the concept node to a user node, while the edge representing a user corresponding to the user node likes the image corresponding to the concept node. The social graph may include an edge connecting the concept node (corresponding to the image) to another concept node comprising a comment made on the image (e.g., by a user of the social-networking system). The social graph may include an edge connecting the concept node (corresponding to the image) to another concept node corresponding to an event that the image is associated with. The social graph may include an edge connecting the concept node (corresponding to the image) to another concept node corresponding to a photo album that the image belongs to. Particular embodiments contemplate any suitable edges that may connect to the concept node corresponding to the image in the social graph. The social-networking system may store one or more edges connecting to the concept node corresponding to the image in one or more data stores of the social-networking system.

In particular embodiments, at step320, the servers may access a data store for data associated with the image. For example, the servers may access one or more data stores storing the concept node corresponding to the image and edges connecting to the concept node (as described above) for data associated with the image such as an author, a location, a time stamp, one or more users tagged in the image, a number of “likes”, one or more comment made on the image, or an album including the image. In some embodiments, the servers may also access the image's metadata for data associated with the image (e.g., a time stamp, a comment). The servers may also access one or more data stores of the social-networking system (or third-party systems) for other content (e.g., text content) of the structured document.

In particular embodiments, at step330, the servers may construct a text string corresponding to the image (e.g., an alternate text for the image). The text string is configured to be audibly read out by the client computing device. In particular embodiments, the servers may construct a text string corresponding to the image in a spoken manner of a natural language. In particular embodiments, the servers may construct the text string based at least in part on at least some of the data associated with the image.

For example, the data associated with the image may include an author of the image (e.g., “John”), a location of the image (e.g., “Grand Canyon”), and7users tagged in the image (e.g., “A”, “B”, “C”, “D”, “E”, “F”, and “G”). Based on the data associated with the image, the servers may construct a text string corresponding to the image in a spoken manner in English such as “This is John's photo with A, B, C, D, E, F, and G. The photo was taken at Grand Canyon.”

In one embodiment, the text string constructed by the servers may also include one or more comments associated with the image. For example, data associated with an image of Arc de Triomphe in Paris may include an author (“Bill”), a user tagged in the image (“Mary”), a location “Arc de Triomphe”, and a comment “Great view from Ave des Champs-Elysees!” The servers may include the comment in a text string corresponding to the image such as “This is Bill's photo with Mary. The photo was taken at Arc de Triomphe, and entitled ‘Great view from Ave des Champs-Elysees!’.”

In particular embodiments, the data associated with the image may correspond to one or more nodes (user nodes or concept nodes) of the social graph. The servers may construct the text string corresponding to the image based on social-graph proximity between each of the nodes and a user node corresponding to the particular user in the social graph.

In particular embodiments, the servers may include only a portion of the data associated with the image that correspond to one or more nodes (user nodes or concept nodes) in the social graph that are within a specified degree of separation from a user node corresponding to the particular user. For example, in the example Grand Canyon image above, users “D” and “E” may be first-degree friends of the particular user, while other users (“A”, “B”, “C”, “F”, and “G”) may be separated from the particular user by two or more degrees in the social graph. The servers may only include first-degree friends of the particular user in the text string such as “This is John's photo with D, E, and five other people. The photo was taken at Grand Canyon.” In this way, only information more relevant to the particular user (i.e., a viewer of the structured document) is included in the text string and configured to be read out audibly to the particular user by the client computing device.

In particular embodiments, the servers may construct the text string corresponding to the image based on social-graph affinity between each of nodes from the data associated with the image and the particular user. For example, in the example Grand Canyon image above, the servers may rank the users “A”, “B”, “C”, “D”, “E”, “F”, and “G” based on their respective affinity coefficients with the particular user. The ranked order of those users may be “E”, “A”, “B”, “D”, “C”, “G”, and “F.” The servers may construct the text string to include top ranked users (e.g., top 3 ranked users) in the text string for the particular user such as “This is John's photo with E, A, B, and four other people. The photo was taken at Grand Canyon.” In this way, only information more affine to the particular user is included in the text string and configured to be read out audibly to the particular user by the client computing device.

In particular embodiments, the servers may construct the text string based on text content of the structured document. The servers may construct the text string based on a language for the text content of the structured document. For example, if some or all of the text content of the structured document is in Spanish, the servers may construct the text string in Spanish, such as “Esta es la foto de John con A, B, C, D, E, F y G. La foto fue tomada en el Grand Canyon.” for the example Grand Canyon image above.

The servers may omit in the text string information that has been mentioned in the other content of the structured document. For example, the structured document may include a photo album (or an event) that includes the example Grand Canyon image above. If the location “Grand Canyon” of the album (thus the location of the image) has been described with the photo album (or event) in the structured document, the servers may construct the text string without the location information (e.g., “This is John's photo with A, B, C, D, E, F, and G”). The servers may construct a null text string corresponding to the image (or skip constructing a text string corresponding to the image), if data associated with the image has been described by text in other portion of the structured document.

In particular embodiments, the servers may construct the text string based on a network state of the client computing device. For example, as the text string corresponding to the image may add to a total size of the structured document to be sent to the client computing device for display, it is desirable to reduce a size of the text string if the client computing device has a network connection that is constrained in bandwidth (e.g., a cellular connection). In particular embodiments, the servers may determine a network state of the client computing device (e.g., a cellular connection, a Wi-Fi connection, or an Ethernet connection). If the client computing device has a network connection that is constrained in bandwidth such as a cellular connection, the servers may construct a shorter version of the text string corresponding to the image included in the structured document.

For example, the servers may construct a shorter version of the text string corresponding to the example Grand Canyon image above by including only some (or none) of the users tagged in the image (e.g., “This is John's photo with D and G. The photo was taken at Grand Canyon.”, or “This is John's photo taken at Grand Canyon.”). The servers may construct a shorter version of the text string by including only some of the users tagged in the image based on social-graph proximity between each of the tagged users and the particular user as described above.

The servers may also construct a shorter version of the text string corresponding to the example Grand Canyon image above by omitting the location information of the image (e.g., “This is John's photo with A, B, C, D, E, F, and G.”).

In one embodiment, the servers may construct a null text string corresponding the image included in the structured document (or skip constructing a text string corresponding to the image) if the client computing device displaying the structured document has a network connection that is constrained in bandwidth.

In particular embodiments, at step340, the servers may send the structured document with the text string to the client computing device for presentation for the particular user. A web browser (or any suitable application configured to display structured documents or web pages) hosted by the client computing device may display the structured document, while a text-to-speech program (or any suitable application) hosted by the client computing device may read out audibly (e.g., through a speaker of the client computing device) text content of the structured document, including the text string corresponding to the image. In one embodiment, the web browser (or another suitable application) may also display the text string adjacent to or overlaying the image (e.g., as a caption for the image) in the structured document. The web browser may display the text string adjacent to or overlaying the image in response to a user event (e.g., a mouse click event, a mouse hover-over event) by the particular user.

Particular embodiments may repeat one or more steps of the method ofFIG. 3, where appropriate. Although this disclosure describes and illustrates particular steps of the method ofFIG. 3as occurring in a particular order, this disclosure contemplates any suitable steps of the method ofFIG. 3occurring in any suitable order. Moreover, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method ofFIG. 3, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method ofFIG. 3.

In particular embodiments, computer system400includes a processor402, memory404, storage406, an input/output (I/O) interface408, a communication interface410, and a bus412. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor402includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor402may retrieve (or fetch) the instructions from an internal register, an internal cache, memory404, or storage406; decode and execute them; and then write one or more results to an internal register, an internal cache, memory404, or storage406. In particular embodiments, processor402may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor402including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor402may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory404or storage406, and the instruction caches may speed up retrieval of those instructions by processor402. Data in the data caches may be copies of data in memory404or storage406for instructions executing at processor402to operate on; the results of previous instructions executed at processor402for access by subsequent instructions executing at processor402or for writing to memory404or storage406; or other suitable data. The data caches may speed up read or write operations by processor402. The TLBs may speed up virtual-address translation for processor402. In particular embodiments, processor402may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor402including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor402may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors402. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In particular embodiments, memory404includes main memory for storing instructions for processor402to execute or data for processor402to operate on. As an example and not by way of limitation, computer system400may load instructions from storage406or another source (such as, for example, another computer system400) to memory404. Processor402may then load the instructions from memory404to an internal register or internal cache. To execute the instructions, processor402may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor402may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor402may then write one or more of those results to memory404. In particular embodiments, processor402executes only instructions in one or more internal registers or internal caches or in memory404(as opposed to storage406or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory404(as opposed to storage406or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor402to memory404. Bus412may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor402and memory404and facilitate accesses to memory404requested by processor402. In particular embodiments, memory404includes random access memory (RAM). This RAM may be volatile memory, where appropriate Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory404may include one or more memories404, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In particular embodiments, storage406includes mass storage for data or instructions. As an example and not by way of limitation, storage406may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage406may include removable or non-removable (or fixed) media, where appropriate. Storage406may be internal or external to computer system400, where appropriate. In particular embodiments, storage406is non-volatile, solid-state memory. In particular embodiments, storage406includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage406taking any suitable physical form. Storage406may include one or more storage control units facilitating communication between processor402and storage406, where appropriate. Where appropriate, storage406may include one or more storages406. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface408includes hardware, software, or both, providing one or more interfaces for communication between computer system400and one or more I/O devices. Computer system400may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system400. As an example and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces408for them. Where appropriate, I/O interface408may include one or more device or software drivers enabling processor402to drive one or more of these I/O devices. I/O interface408may include one or more I/O interfaces408, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.